dcsimg

Slicing and Dicing on the Command Line

If you don't know text, you don't know Linux. There are a host of methods for reformatting plain text -- including the text used by graphical applications like spreadsheets and email programs.

Plain text is a series of characters delimited into lines by newline (LF, line feed) characters. You can send this text directly to a terminal window with a utility like cat(1). There are no hidden formatting codes; it’s “just the text, ma’am.”

Before the puns get any worse, let’s dig in!

Quick Review

As you saw in last month’s column (if you didn’t see the column, you might want to review it), to start a new line at any point in plain text, simply insert a newline character. To join two lines, remove the newline between them — and maybe add a space or TAB character to separate them.

When a terminal or printer reads a TAB character, it moves the current position to the next tabstop. TAB characters are also used as field separators; you can make a simple database with TABs between the fields and a newline at the end of each record.

Linux utilities can also reformat text that doesn’t contain TABs. We’ll see examples of that, too.

Lots of Possibilities

Many GNU utilities started in the days of Unix — back when a tty really was a teletype. Without a graphical display (or a graphical editor) to rearrange text, programmers came up with many ways to slice, dice, and reassemble data from scripts and the command line.

We’ll see some of those ways: Enough ways, I hope, that people new to this way of handling text will be ready to find other ways — and gurus will still get a few surprises.

Starting with a Spreadsheet

Plain text can come from lots of places, including:

  • The output of a utility (grep, for instance),
  • Text saved from an application (see Figure One for an example),
  • Text pasted into a terminal window from a graphical application, as in Figure Two near the end of this article.

Note that some of this text may not be “plain” characters. For instance, if you’re copying from a web page designed by a Macintosh user, the designer may have unwittingly included the Macintosh encoding of a special character (maybe a “curly quote”) that isn’t recognized on your Linux system.

For the first few examples, let’s use an OpenOffice.org spreadsheet file saved as plain text. (On the File menu, choose Save As, type Text CSV.) Assuming that the data doesn’t contain any TAB characters, you can set the Field Delimiter to TAB and the Text Delimiter to none (delete the default quote mark in that dialog box). Figure One shows this.

Figure One: Saving a spreadsheet as plain text
Figure One: Saving a spreadsheet as plain text

Below are are two views of the resulting file data.txt (renamed from the default data.csv). First, plain cat outputs the TAB characters between fields, which the terminal displays by moving to the next tabstop position. Next, cat -tve shows what’s actually in the file:

$ cat data.txt
STATE	CITY	COUNTY	POP.	GOVT.
AZ	Ely	Gila	123	Mayor
CA	Alma	Lolo	345	Sheriff
TX	Leroy	El Paso	22	Bubba
$ cat -tve data.txt
STATE^ICITY^ICOUNTY^IPOP.^IGOVT.$
AZ^IEly^IGila^I123^IMayor$
CA^IAlma^ILolo^I345^ISheriff$
TX^ILeroy^IEl Paso^I22^IBubba$

Checking the data file with cat -tve or od -c is a good idea. They’ll reveal “hidden” or “non-plain” characters buried in the data. Notice the space character in the field El Paso. Because the field separator is a TAB, the space doesn’t cause any problems.

Utilities that Understand TABs

Scripting languages (Perl, awk, …) can parse and write TAB-separated data. Table One lists some other Linux utilities that handle TABs.

Table One: Some utilities that understand TABs

Utility Description
cut(1) Remove sections from each line of files
echo(1), printf(1) Write arguments to standard output (\t makes a TAB)
expand(1), unexpand(1) Convert TABs to spaces, spaces to TABs
paste(1) Merge lines of files into TAB-separated output
sed(1) Stream editor
sort(1) Sort data by one or more of its fields

Whether your data comes from a spreadsheet or some other source, if you can massage your data into TAB-separated fields, the examples below can help you slice and dice it. Examples toward the end of the article cover other types of data.

Comments on "Slicing and Dicing on the Command Line"

We came across a cool web-site that you simply may well delight in. Take a look when you want.

Thanks a lot for the article post.Really looking forward to read more. Great.

Here are a few of the web pages we advise for our visitors.

Here is a superb Blog You may Discover Intriguing that we encourage you to visit.

Just beneath, are a lot of entirely not related internet sites to ours, even so, they may be certainly worth going over.

The time to study or go to the content material or web sites we have linked to below.

The info mentioned in the write-up are several of the most beneficial obtainable.

Very handful of web-sites that happen to be comprehensive beneath, from our point of view are undoubtedly nicely really worth checking out.

We came across a cool website that you could possibly appreciate. Take a search in case you want.

“This is one awesome blog post.Really thank you! Want more.”

Wonderful story, reckoned we could combine a handful of unrelated information, nonetheless definitely really worth taking a appear, whoa did 1 learn about Mid East has got much more problerms at the same time.

“I like the helpful info you provide in your articles. I??™ll bookmark your blog and check again here regularly. I’m quite sure I will learn many new stuff right here! Best of luck for the next!”

“There may be noticeably a bundle to find out about this. I assume you made sure good points in options also.”

“Really appreciate you sharing this blog post.Much thanks again. Keep writing.”

“One other thing to point out is that an online business administration training is designed for students to be able to smoothly proceed to bachelors degree programs. The Ninety credit certification meets the other bachelor diploma requirements and once you earn the associate of arts in BA online, you may have access to up to date technologies on this field. Some reasons why students need to get their associate degree in business is because they may be interested in this area and want to obtain the general instruction necessary ahead of jumping right bachelor education program. Thx for the tips you actually provide inside your blog.”

“I think that a property foreclosures can have a important effect on the client’s life. Mortgage foreclosures can have a 8 to few years negative impact on a borrower’s credit report. Any borrower who have applied for a home loan or virtually any loans for instance, knows that a worse credit rating can be, the more hard it is to obtain a decent financial loan. In addition, it can affect a new borrower’s power to find a good place to let or rent, if that results in being the alternative housing solution. Good blog post.”

“A round of applause for your article.Really looking forward to read more. Really Great.”

“Thank you ever so for you blog post.Really looking forward to read more. Really Great.”

“Wow, this paragraph is good, my sister is analyzing these kinds of things, thus I am going to inform her.”

“Thank you for sharing most of these wonderful blogposts. In addition, the right travel along with medical insurance program can often ease those considerations that come with visiting abroad. Any medical crisis can quickly become expensive and that’s bound to quickly slam a financial burden on the family finances. Setting up in place the great travel insurance bundle prior to setting off is worth the time and effort. Cheers”

“Hi there, I discovered your web site by way of Google even as searching for a similar topic, your website got here up, it appears good. I have bookmarked it in my google bookmarks.”

“There is definately a great deal to learn about this subject. I love all of the points you made.”

“Very informative article post. Much obliged.”

The details mentioned in the write-up are some of the best accessible.

“Hi! I’m at work surfing around your blog from my new apple iphone! Just wanted to say I love reading through your blog and look forward to all your posts! Carry on the outstanding work!”

“This is one awesome blog article.Really looking forward to read more. Awesome.”

“My brother suggested I might like this website. He was entirely right. This post truly made my day. You cann’t imagine simply how much time I had spent for this info! Thanks!”

“This article will assist the internet users for creating new webpage or even a weblog from start to end.”

“This design is steller! You certainly know how to keep a reader amused. Between your wit and your videos, I was almost moved to start my own blog (well, almost…HaHa!) Wonderful job. I really loved what you had to say, and more than that, how you presented it. Too cool!”

“Heya! I’m at work browsing your blog from my new iphone! Just wanted to say I love reading your blog and look forward to all your posts! Keep up the great work!”

I am so grateful for your blog post.Thanks Again.

“It’s very easy to find out any topic on net as compared to books, as I found this paragraph at this site.”

One of our visitors just lately advised the following website.

What’s up, this weekend is nice in support of me, because this moment i am reading
this fantastic informative article here at my residence.

Sites of interest we have a link to.

That is the finish of this post. Right here you will obtain some web sites that we think you will appreciate, just click the links.

Just beneath, are numerous absolutely not related web sites to ours, even so, they’re surely worth going over.

Here are several of the internet sites we advise for our visitors.

Here is a superb Blog You might Discover Intriguing that we encourage you to visit.

Always a large fan of linking to bloggers that I adore but don?t get quite a bit of link like from.

That is the end of this post. Here you will find some websites that we think you?ll enjoy, just click the hyperlinks.

The time to study or take a look at the content or websites we’ve linked to below.

Check below, are some completely unrelated internet sites to ours, nevertheless, they are most trustworthy sources that we use.

Please visit the sites we follow, which includes this one particular, because it represents our picks from the web.

Here is a good Weblog You might Find Exciting that we encourage you to visit.

Every the moment inside a though we select blogs that we study. Listed beneath are the most recent sites that we pick out.

Here are a few of the sites we advocate for our visitors.

We like to honor a lot of other web sites around the internet, even though they aren?t linked to us, by linking to them. Under are some webpages really worth checking out.

Leave a Reply