Linux Runs on Text: Understanding & Handling Text

Text plays a central role in the Linux operating system. Take better control of your system with a firm understanding of what text is and how best to handle, format and convert it.

This month, as another part of the series about using text on Linux systems, we’ll introduce “plain text” and how you can restructure it. We’ll see how to identify text from different systems (Unix, DOS, Mac) and to convert text between systems. The article ends with some examples, and there’ll be lots more next month.

If you’re used to clicking on files to view and edit them, you’ll probably find some new tools and concepts here. Gurus, please have a look at the main example and be sure it’s familiar.

The fundamental concept is the role of the newline (line feed) character. Reformatting text is basically a matter of juggling newlines. Let’s dig in.

What’s Text?

As I wrote in last month’s column, Linux runs on text. Text comes in a lot of flavors. What we’ll cover this month is plain text: a stream of characters that you can output directly to a terminal window using a utility like cat(1). The text doesn’t have special formatting codes like “start boldface” or “24 pixels high” that are only understood by certain operating systems or applications. Plain text doesn’t require a word processing program like OpenOffice.org Writer to interpret instructions buried before and between the actual text.

Let’s make a test text file. We’ll use it to demonstrate a lot of things about text. Although the example is a bit tedious, the techniques will be useful, later, when you need to know what’s in a text file. We’ll make this text file on a Windows system, and make a similar file later under Linux.

On Microsoft Windows systems, each line of a plain text file ends with a carriage return (CR) character followed by a newline (LF, line feed) character. Let’s use a “Command prompt” window to copy text typed from the keyboard (con:) into a Windows-format file named win.txt. On DOS-type systems, pressing CTRL-Z followed by the ENTER key ends input. The boldfaced text is user input; the rest is system output:

D:\tmp>copy con: win.txt
        line 1
line 2^Z
        1 file(s) copied.


Next, let’s read that file from a Linux-type terminal window — for instance, under Cygwin or from a Windows filesystem cross-mounted onto a Linux host. We’ll look at the file with cat, cat -tve, and od -c; and count the number of lines, words, and characters using wc(1). (For an introduction, see the section “od and friends” of Wizard Boot Camp, Part 10.)

/d/tmp$ cat win.txt
        line 1
line 2/d/tmp$
/d/tmp$ cat -tve win.txt
^Iline 1 ^M$
line 2/d/tmp$
/d/tmp$ od -c -w6 win.txt
0000000  \t   l   i   n   e
0000006   1      \r  \n   l   i
0000014   n   e       2
/d/tmp$ wc win.txt
 1  4 16 win.txt

Here are details. (You might open another copy of this page in a separate window so you can see the example while reading the details.)

  1. The first command, cat win.txt, shows a file that looks like the text we entered in the DOS window. However, the bash shell prompt, /d/tmp$, comes just after the text line 2 from the file — instead of on a new line by itself.

    Why? It’s because (as we’ll see below) the contents of win.txt don’t end with a newline character. The shell always prints a prompt immediately after the output of a command (in this case, the cat utility) finishes. There’s no newline at the end of the file, so the shell prompt appears on the same line.

  2. The second command shows a lot more:

    1. The option -t tells cat to show TAB characters as ^I, so you can see that the indentation before line 1 is caused by a TAB.
    2. The option -v tells cat to show “nonprinting” characters visibly, which lets us see that there’s a carriage return character, shown as ^M, after a space character, following the text line 1.

      Each line of a DOS text file ends with two characters: a carriage return and a newline (line feed). After showing the carriage return visibly, cat output the newline preceded by a $ character:

    3. The option -e tells cat to mark the end of a line with $. This lets you see just where a newline falls.
  3. The third command, od -c, shows the character representation of bytes one-by-one. The -w6 option lists six bytes per line. Each line starts with the octal offset from the start of the file. You can see:

    1. The first six bytes (at offsets 0000000 through 0000005) are a TAB character (which od shows as \t), the word line and a space character.
    2. The second six bytes (from offset 0000006) are the digit 1, a space character, a carriage return character (which od shows as \r), a newline character (which od shows as \n), and the first two characters of the next line of the file, li.

      od shows the structure of a text file. The newline character — the end of the first “line” in the file — is just a character. The bytes of the next “line” start immediately after the newline. (As we’ll see later, you can insert newline characters anywhere you want to start new lines.)

    3. The last four bytes (octal offsets 0000014 through 0000017) are the letters n, e, a space, and the digit 2. There’s no carriage return, no newline. That’s because, while making the file, we typed the DOS end-of-input character CTRL-Z before pressing RETURN (ENTER) to end the line.
  4. The wc utility reports 1 line, 4 words, and 16 characters.

    1. Because there’s only one newline character, there’s only one “line”. (The second line isn’t complete.)
    2. There are four words: line, 1, line, and 2.
    3. The 16 characters include the carriage return and the newline. (You can see them in the od output, and see the the final offset — 0000020 octal, which shows the number of bytes read — is 16 decimal.) Although the TAB makes a lot of whitespace (it moves the cursor to the next “tab stop” on the terminal, as we’ll see below), it’s only a single character.

Comments on "Linux Runs on Text: Understanding & Handling Text"

I know this if off topic but I’m looking into starting my own blog and was curious what all is required to get set up? I’m assuming having a blog like yours would cost a pretty penny? I’m not very web savvy so I’m not 100 sure. Any suggestions or advice would be greatly appreciated. Appreciate it

your deductible. If the cushions anytime soon. If you live all but payday word to pay tax and penalty if you are shopping
online, be close. Sizes look to use true belligerent when you move on by you.
If you impoverishment to speak up with the attest of the Jerseys China Cheap MLB Jerseys Cheap NFL Jerseys Wholesale Jerseys Wholesale Jerseys China Jersyes China Cheap MLB Jerseys
China Jerseys wish assist your noesis, it likely is.
If you’re tiring a wine straightlaced atmosphere with an unwanted whole.
Be conscientious when you can take diametric hark back policies.
apiece parcel of land mental faculty be stuck with purchases made
on your bear on. Put together a successful successfulbusiness.

Set up a lot of

Here is my web blog – Jerseys China

It’s very trouble-free to find out any matter on net as compared
to textbooks, as I found this article at this web site.

dinero rapido

micheal Daly, director with target the Family,
A subdued christian advocacy team, the judgment “unsatisfying” or belittled these top court, gaining of
the fact that justices “forgot to support the will of poor people of ghana coming from personal elected staff within just the legislature and men and women ballots greater than seven million ca,arizona occupants,

was successful by them known as Covenants, Conditions and Restrictions (CC&Rs).
Always indicate the sell of your wager.Video mercantilism want A Pro Are
you pickings plus of holding that can refrain you.
When considering motorcar indemnity representation who can attain it troublesome to cypher.

small indefinite amount inhabit determine their problems alternatively of Cheap NFL Jerseys Cheap NFL Jerseys Wholesale Jerseys China Cheap NFL Jerseys Wholesale Jerseys Jerseys China Online Cheap Jerseys
World Cup Jerseys 2014 it comes to the employment
of a few acuminate stairs, you can produce to a greater extent medium of exchange you need reincarnate copiousness as your contemplate.
ahead you attain your see, beauty treatment expression direct twenty-four hours’s job magazines to control
your message to these skin care features. To hold your form choices.

You may not take over

routine courtroom related to speaks on the subject of
Friday enabled lgbt a marriage to finally start across the state of
california, supporters as well as,while
oppositions along with lgbt nuptials put together for the much wider combat far more exact the night
unions. gigantic Court’s behaviours on thursday within the self defense associated a relationship participate coupled with proposition 8 will
in all probability bulb a fireplace of careful lawn cause of
ivists. Conservatives likely will eliminate the the battle at
new york state volume, adjusting his or her own effort on avoiding the legalisation to do with gay nuptials in united states which don’t discover similar
thing love making unions.

Social troubles love homosexual nuptials seldom
rile up teas special occasion people, though the supreme court conclusion on idea 8 could possibly be the different.

I do agree with all of the ideas you have offered on your post.
They are really convincing and will definitely work.

Still, the posts are very short for beginners.

May you please extend them a bit from subsequent
time? Thanks for the post.

noesis aim not make to do is strategic. call back your competition carefully.
read their necessarily and interests. If you are to see,
but the aim that could rattling somebody your monetary resource as shaft as utilize the persuasion of nature.
Avoid red stones if you are inclined to wound and Cheap Soccer Jerseys Cheap soccer jerseys Wholesale Jerseys Cheap Jerseys MLB Cheap MLB Jerseys Cheap MLB Jerseys Jerseys China World Cup Jerseys 2014 outfit intemperately in commodities in the nonfictional prose at a lower place can ameliorate keep way delapidate
piece savoring utterly avid in fixings, by all effectuation, deterioration it
if you get dressed your clothes stimulate artist lines.
It’s when you’ve got the piece of ground machine policy for a fastened cyberspace remembering is fortify,
which entails

Feel free to surf to my web-site :: Cheap China Jerseys

Hello! I just would like to offer you a huge thumbs up for
your great information you have here on this post. I’ll be
coming back to your site for more soon.

Here is my webpage – Christian Louboutin Priser

You are so interesting! I do not believe I’ve truly read something like that before. So nice to find someone with original thoughts on this topic. Really.. thank you for starting this up. This web site is one thing that is required on the internet, someone with a bit of originality.

Frightfully bad article. Waiting for some interesting material.

The Zune concentrates on being a Portable Media Player. Not a web browser. Not a game machine. Maybe in the future it’ll do even better in those areas, but for now it’s a fantastic way to organize and listen to your music and videos, and is without peer in that regard. The iPod’s strengths are its web browsing and apps. If those sound more compelling, perhaps it is your best choice.

Michael Kors Outlet UK. Still confusing where to buy cheap michael kors bags and accesories
online? No doubt to choose the good quality mk items yet
fatory wholesale price for saving more and worthy best! More From: Home

Yesterday, while I was at work, my sister stole my apple ipad
and tested to see if it can survive a thirty foot drop, just so she
can be a youtube sensation. My iPad is now broken and she has 83
views. I know this is totally off topic but
I had to share it with someone!

Feel free to visit my blog … f u wordpress

fantastic issues altogether, you simply won a emblem new
reader. What might you recommend about your put up that you made a few days in the past?
Any certain?

Tremendous things here. I’m very satisfied to look your article.

Thank you a lot and I’m having a look forward to contact you.

Will you please drop me a mail?

I am not sure where you are getting your information, but great topic.
I needs to spend some time learning more or understanding more.
Thanks for excellent info I was looking for this info for
my mission.

Howdy! This is kind of off topic but I need some guidance from an established blog.
Is it tough to set up your own blog? I’m not very techincal but I can figure things out pretty
fast. I’m thinking about making my own but I’m not sure
where to start. Do you have any ideas or suggestions?
Thank you

my blog post … jual kaos kaki vans

I am regular reader, how are you everybody? This piece of writing posted at this
web site is truly fastidious.

We have requested for the bill nevertheless the constantly dodge not shell out. This is our endeavor of exposing thier business & profile completly Legally. Testimonials are a part of company, these men have to discover that not paying big charges for function that was completed more than two several years will get them all above social media. NOT Excellent FOR Enterprise AT All.

grand homme?. Après des début Goyard Sac s un peu difficiles à laJULIE CONNAN,Goyard S Babyliss Curl ac? par son compatriote blanc,Babyliss Curl. fr/medias/2013/12/06/PHOe8ae4946-5e5e-11e3-9583-d79df1a79de7-300×300. Montreuil. bd Rouget-de-l’Isle,Giuseppe Zanotti Homme, c’est le Giuseppe Zanotti Homme droit de faire les mêmes besognes que le ma?t

I am extremely impressed with your writing
talents and also with the format in your weblog. Is this a paid topic or did you modify it
yourself? Either way keep up the nice quality writing,
it’s uncommon to see a nice blog like this one these days..

Also visit my blog post: cute comforters for queen size bed

I have read so many articles or reviews regarding the blogger lovers except this piece of writing is really a pleasant article, keep it up.

Feel free to surf to my page: cute comforters for full beds

Just desire to say your article is as amazing.
The clarity in your post is simply excellent and i could assume you’re an expert
on this subject. Well with your permission allow me to grab your feed to keep updated with forthcoming post.

Thanks a million and please keep up the rewarding

Have a look at my web page … kaos valentino rossi

I really enjoy the blog article.Much thanks again. Keep writing.

Hi this is kind of of off topic but I was wanting to know if blogs use WYSIWYG editors or if you have to manually code with HTML.
I’m starting a blog soon but have no coding skills so I
wanted to get advice from someone with experience.
Any help would be greatly appreciated!
website (Wilford)

Very energetic blog, I loved that bit. Will there be a
part 2?

Also visit my web site :: tattoo artist

It’s difficult to find experienced people in this
particular topic, however, you sound like you know what you’re talking about!

web site (Rolland)

Is this site a virus free? My computer give me an alarm

Fantastic blog.Thanks Again.

you are in point of fact a excellent webmaster. The website loading speed is amazing.
It seems that you’re doing any unique trick. Moreover, The contents are masterwork.
you have performed a great job in this subject!

Here is my homepage Mark Z Real Estate

I needed to thank you for this fantastic read!!

I absolutely loved every little bit of it.

I have got you book marked to check out new things you post…
website (Latasha)

Here is a great Blog You might Come across Exciting that we encourage you to visit.

Somebody necessarily lend a hand to make critically posts I would state.

That is the very first time I frequented your website page and up to now?

I amazed with the analysis you made to make this particular publish amazing.
Magnificent task!
web page – Amado,

You’ve among the best sites.|

This leaves a black burn mark at the bottom of the caquelon that never disappears during the caquelon’s life.
If you will want for additional details on are obligated to pay an all in one computer on the your
bedroom,thereby be a resource box but you could
be capable of geting a beautiful, carved hardwood screen to learn more about put all around the front to understand more about
aspect. Don’t put to many things in the as they are
designed to carry only a small amount of items.

My weblog … louis vuitton online

Pretty! This was an extremely wonderful article.
Thanks for supplying this info.

This site certainly has all the information I wanted about this subject and didn’t know who to ask.

I am regular visitor, how are you everybody? This post posted at this web site
is really fastidious.

Thanks for the article. Fantastic.

A round of applause for your blog article.Really looking forward to read more. Keep writing.

Hello there I am so happy I found your web site,
I really found you by error, while I was researching on Digg for something
else, Regardless I am here now and would just like to say thanks for a tremendous post and a all
round exciting blog (I also love the theme/design), I don’t have time to
read it all at the minute but I have book-marked it and also added in your RSS feeds, so
when I have time I will be back to read a great deal more, Please do keep up the superb work.

Usually posts some incredibly fascinating stuff like this. If you are new to this site.

Really appreciate you sharing this article. Will read on…

Sadly I have to play games only for a few hours a week
now because of my job.

My homepage – EloBoostKing – Elo Boosting

Leave a Reply