dcsimg

Wizard Boot Camp, Part 10: Utilities You Should Know

The time has come to leave Hogwarts, young wizard! We wrap up our ten-part series on becoming a command-line wizard with a look at more utilities you should know.

We wrap up the Wizard Boot Camp series with a third and final article about utility programs that you should know about — and some not-so-obvious ways to use them.

csplit, split

Some large files — including archives, multipart email messages, business information for a large set of customers, and data files with repeating patterns — may need to be split into smaller “chunks” for reorganization, easier storage or transport. That’s what csplit(1) is for. Give it a pattern, an offset, a repetition count, and/or a line number, and it will parse an input file into a series of smaller output files.

By default, csplit‘s output files are named xx00, xx01, and so on. To rearrange a big file, you can simply cat those output files together in a different order. Here’s a simple example using the shells’ curly-brace operators:

$ csplit oldfile '/pattern/' '{5}'
  ...creates files xx00 - xx05...
$ cat xx0{5,0,2,1,3,4} > newfile
$ rm xx0[0-5]

Using a shell wildcard operator, like xx0[502134], wouldn’t work with cat because it sorts filenames in alphanumeric order. We’ll see more about csplit in a future column — part of a new series about handling text files.

In these days of multi-gigabyte files and terabyte (or larger) filesystems, a file can still be “too big.” For instance, if you’re trying to send a huge file over a network connection that often times out, even the automatic-retry ability of some file transfer utilities won’t get all of the file through.

Splitting the file into smaller chunks, then reassembling the pieces on the other end, can save headaches and time. The split utility is great for that. It splits input — either files or stdin — into equal-sized files named (by default) xaa, xab, xac, and so on. (The last file may be smaller.) For instance, on the sending side, split the file into 1-Megabyte chunks:

$ ls -l kcpr.mp3
-rw-r--r-- ... 70535208 ... kcpr.mp3
$ split -b1m kcpr.mp3
$ ls -l x??
-rw-r--r-- ...  1048576 ... xaa
-rw-r--r-- ...  1048576 ... xab
...
-rw-r--r-- ...   280616 ... xcp

After you’ve transmitted all of the files, do a quick check on the receiving side to be sure they all have the same size. Then cat them together.

% ls -l x??
-rw-r--r-- ...  1048576 ... xaa
...
-rw-r--r-- ...   280616 ... xcp
% cat x?? > kcpr.mp3
% rm x??

Using a checksum program like md5sum on both the original file and the reconstructed version can give you more confidence.

file

In general, Linux doesn’t require filename extensions such as .exe or .txt to know what to do with a file. Executable files — files whose execute bit was set by, for instance, chmod — start with a two-byte magic number. The best-known magic number is probably #!, which lets you specify the file’s interpreter (/bin/sh, /usr/bin/perl, etc.)

The file utility will guess what type of data is in many types of files. That includes unidentified or mis-identified files you receive attached to an email message. For example:

$ file mystery-file.dat
mystery-file.dat: PDF document, version 1.3

locate, updatedb

If your system runs the updatedb utility (from cron or otherwise), you’ll have a database on your system that lets you find files or directories by name much faster than a command like find / -name .... The locate utility searches that database. For instance, to find all files and directories whose name includes examples:

$ locate examples
/usr/bin/dh_installexamples
/doc/adduser/examples/INSTALL
/doc/zsh/compctl-examples.gz
...

The locate manpage has details — including how to use wildcard characters to restrict matching. But some uses aren’t quite so obvious. If you think about the syntax of the pathnames you want, you can often use string-matching to find them. To locate any file in a directory named examples, for instance, use a search pattern that matches a directory name in a pathname:

locate /examples/

To do more sophisticated pattern-matching, you can filter locate‘s output through a tool like grep. You can even dump the entire database:

locate / | grep ...

To search the contents of files or directories with a certain name, pipe the list of pathnames to xargs grep. For instance, to search all files with a name including foo for text containing bar, try this:

$ locate -0 foo | xargs -0 grep -Hs bar | cat -v
/usr/local/bin/ascript:echo "back at the bar..."
/usr/share/doc/m4/examples/foo:bar
Binary file /usr/share/emacs/21.4/lisp/mail/footnote.elc matches

We’re using the -0 (zero) option for both locate and xargs; it separates pathnames with NUL characters, which avoids problems caused by “special” characters in filenames. The grep option -H makes tells grep to always output a filename (even if xargs happens to pass only one filename to grep). And the grep option -s keeps grep silent about arguments that are directory files or unreadable files. The cat -v avoids sending “unprintable” characters to your terminal.

You may want to run updatedb multiple times to make more than one locate database: one for all users, one for system files, one for each user’s home directory (readable only by that user), and so on. In that case, you and other users may want to set the LOCATE_PATH environment variable to tell locate which databases to search:

$ grep LOCATE_PATH /etc/profile
LOCATE_PATH="/var/cache/locate/locatedb:/usr/local/locatedb/$USER"
export LOCATE_PATH MANPATH PATH

look

Everyone knows about the grep utilities. Less well-known is look. It searches the first characters on each line of data — like a grep search starting with the anchor character ^ (caret). look defaults to a linear (sequential) search or uses binary search with its -b option. A binary search rapidly searches a sorted data file — even a very large one.

By default, look searches the system word list. That’s handy for checking spelling. If there are words that start with the argument you type, you’ll see them:

$ time look gas
Ga's
Gascony
Gascony's
gas
gas's
...

real    0m0.088s
user    0m0.065s
sys     0m0.009s

Comments on "Wizard Boot Camp, Part 10: Utilities You Should Know"

It’s very trouble-free too find out any matter on nett ass compared to textbooks,
as I found this piece of writing at thks web site.

Here is my web page where buy cialis prescription

I have learn some good stff here. Definitedly value bookmarking for
revisiting. I wonder how a lot effort you place to crewate this sort of wondedful informative website.

My blog post: herbal cialis

Excellent post. Keepp posting such kind of information on your blog.
Im really impressed by it.
Helllo there, You have done an incredible job. I will certainly digg it and individually suggest to
my friends. I’m sur they’ll be benefited frokm this website.

Also visit my website cialis daily use reviews

Very nice article, exactly what I needed.

Here is my wweb page: cialis trial

Yes! Finally something about generic cialis lowest price.

Here iss my web blog; where buy cialis singapore

Hi there, just became alert to your blog through Google, and fouund that it’s truly informative.
I am going to watch out for brussels. I’ll be grateful if you continue this in future.

Numerous people will bee benefited from your writing.
Cheers!

Look at my webpage – cialis generic 5mg

Inspiring quest there. What happened after? Thanks!

Feel free to surf to my blog post :: discount cialis

I absolutely love your blog.. Great colors & theme. Did you make this amazing
site yourself? Please repoy back as I’m hoping to create my very own blog and would like to learn where you got
this from or juust what the theme is named. Thank you!

My blog; canada pharmacy cialis cost

My brother suggested I may like this website.
He was once totally right. This put up truly made my day.
You can not imagine just how so much time I had spent for this info!
Thank you!

Here is my webpage :: cialis canada

all the time i used to read smaller articles or reviews that
also clear their motive, and that is also happening with this paragraph which I amm reading at this place.

Here is my blog post; cilias in canada on the prices

Nice weblog right here! Also your website loads up very fast!
What web host are you the use of? Can I am getting your associate link in your host?
I wish my site loaded up as fast as yours
lol

continuously i used to read smaller content that also clear their motive, and that is also happening with
this paragraph which I am reading here.

Here is my webpage … canadian pharchamacy

Now I am going away to do my breakfast, after having my breakfast coming over again to read other news.

Thank you for the sensible critique. Me and my neighbor were just preparing to do a little research about this. We got a grab a book from our local library but I think I learned more from this post. I am very glad to see such great information being shared freely out there.

I know this if off topic but I’m looking into starting my own weblog
and was wondering what all is required to get setup?
I’m assuming having a blog like yours would cost a pretty penny?
I’m not very internet savvy so I’m not 100% certain. Any
recommendations or advice would be greatly appreciated. Thank you

Hey this is kinda of off topic buut I was wondering if
blogs use WYSIWYG editors or if yoou have to manually code with
HTML. I’m starting a blog soon but have noo coding skills
so I wated to get guidance from someonne with experience.
Any help would be greatly appreciated!

my website ajanta pharma dahej

Tremendous issues here. I am very satisfied to look your article.
Thank you so much and I am taking a look forward to contact you.
Will you kindly drop me a e-mail?

Also visit my page is prescription needed for cialis canada

I loved as much as you will receive carried out right here.

The sketch is attractive, your authored subject matter stylish.
nonetheless, you command get bought an nervousness over that you wish be delivering the following.
unwell unquestionably come more formerly again since exactly the same nearly very often inside case you shield
this hike.

my webpage; online pharmacy russia

Hi, I do think this is a great website. I stumbledupon it ;) I will revisit yet again since I book marked it.
Money and freedom is the best way to change, may you be rich and
continue to help others.

Here is my weblog: 5mg cialis from canada

It’s a pity you don’t have a donate button! I’d most certainly donate
to this brilliant blog! I suppose for now i’ll settle for bookmarking and adding your RSS feed to my Google account.

I look forward to brand new updates and will talk about this blog with my Facebook group.
Talk soon!

Here is my weblog http://Www.mycanadian-pharmacy.com

My programmer is trying to persuade me to move to .net from
PHP. I have always disliked the idea because of the expenses.
But he’s tryiong none the less. I’ve been using Movable-type
on various websites for about a year and am nervous about switching to another platform.

I have heard good things about blogengine.net. Is there a way I can import all my wordpress posts into it?

Any help would be really appreciated!

my homepage – canadain cialis genaric

Ahaa, its pleasant discussion on the topic of this post here at
this weblog, I have read all that, so now me also commenting at this place.

Feel free to visit my web page – cialys online

If you are going for best contents like myself, just visit this web site all the time since
it gives quality contents, thanks

Review my blog; Prescription meds online

Having read this I believed it was very enlightening.

I appreciate you finding the time and energy to put this content
together. I once again find myself spending a lot of time
both reading and leaving comments. But so what, it was still worthwhile!

Look at my blog – generic tadalafil

Heya i’m for the first time here. I found this board and I find It really useful & it helped me out a lot.
I am hoping to give something back and aid others
such as you helped me.

Stop by my weblog: Canadianpharmacyvi.com

I really like your blog.. very nice colors & theme.
Did you make this website yourself or did you hire someone to do it for you?
Plz reply as I’m looking to design my own blog and would like
to know where u got this from. many thanks

My page … Best buy pill cialis professional erectile dysfunction

Hi there to all, the contents existing at this site are in fact awesome for people experience, well, keep up the nice work fellows.

Also visit my site; Viagra vs sildenefil

It’s difficult to find experienced people for this subject, but you seem like you
know what you’re talking about! Thanks

My web blog: Order coalis on line canada

I always emailed this blog post page to all my associates, because if like to read it next my contacts will
too.

my site :: viagra drop pic price pak

Hello, I wish for to subscribe for this blog to take
most recent updates, therefore where can i do it please assist.

Here is my web blog … pharmacy rx one 60 mg cialis

Amazing things here. I am very happy to peer your article.
Thank you a lot and I am taking a look forward to touch you.
Will you please drop me a e-mail?

Visit my webpage: cialis canada order

Great items from you, man. I have remember your stuff previous
to and you are just too wonderful. I actually like what you have bought right here, certainly like what you are stating and
the way in which through which you are saying it. You make it
entertaining and you continue to care for to stay it wise.
I can not wait to read far more from you. That is really a great site.

my blog … cialis daily 5mg

What’s Going down i am new to this, I stumbled upon this I’ve discovered It absolutely
useful and it has aided me out loads. I’m hoping to give a contribution & assist other users like its aided me.
Good job.

Here is my page; Tadalafil canada

Hey! Quick question that’s completely off topic.
Do you know how to make your site mobile friendly? My web site looks weird when viewing from my iphone 4.
I’m trying to find a theme or plugin that
might be able to fix this issue. If you have any recommendations, please
share. Cheers!

my web-site – http://www.mycanadian-pharmacy.com

Remarkable issues here. I’m very happy to look your post.
Thanks a lot and I am taking a look forward to touch
you. Will you kindly drop me a mail?

Also visit my web-site Cellialis

Hi to all, since I am truly eager of reading this web site’s
post to be updated daily. It contains fastidious
data.

Also visit my blog post :: Canadian viagra via paypal

I would like to thank you for the efforts you’ve put
in penning this website. I really hope to check out the same high-grade blog posts by you in the future as
well. In truth, your creative writing abilities has motivated me to get my very own blog now ;)

Feel free to surf to my site … Cilials pills

It’s very simple to find out any topic on net as compared to books, as I found this
paragraph at this web page.

Here is my weblog … Cialis.com – erectile pills

Excellent blog here! Also your web site loads up fast!
What host are you using? Can I get your affiliate link
to your host? I wish my web site loaded up as quickly as yours lol

Stop by my blog post; Tadalafil tablets prices usa

Hi there colleagues, how is everything, and what you want to say concerning this piece of
writing, in my view its genuinely awesome for me.

Feel free to visit my blog :: cialis sold in canada

My brother recommended I might like this web site.
He was once totally right. This put up truly made
my day. You can not consider simply how much time I had spent for
this information! Thanks!

My webpage: Canada cheap online pharmacy

Way cool! Some very valid points! I appreciate you penning this article and the
rest of the website is really good.

My blog post :: Quick buy market viagra in canada

Very handful of internet sites that transpire to be comprehensive below, from our point of view are undoubtedly effectively worth checking out.

Awesome! Its in fact remarkable article, I have got much clear idea concerning
from this piece of writing.

Also visit my homepage … Buy cialis online

Great article! We are linking to this particularly great post on our
website. Keep up the good writing.

Feel free to surf to my web blog – cheap generic viagra

Leave a Reply