As a user of OpenSolaris, one of the best ways you can contribute to the community is by taking the time to submit the bugs you find in the product. You may also find that you get better traction with OpenSolaris engineering than raising the same issue in the forums - bug counts are reviewed weekly, forum postings are not.

The OpenSolaris project uses Bugzilla as its tracking system and its home page is at http://defect.opensolaris.org. Note, it uses a separate registration database then opensolaris.org, so you'll have to create a new account.

Before filing a bug you should first search to see if it's already being tracked. If so, feel free to add additional information or just CC yourself on the bug to keep informed of its progress. For example, I've CC'd myself on over 140 bugs.

The advanced search form is very powerful, which also makes it somewhat daunting. I mostly use the simple text based search form on the home page. Here are some basic tips I've found helpful for getting the most out of the simple text search:

• By default, only open bugs are returned. Since the problem you are looking for might very well have already been fixed in a development build, prefix your search with ALL, for example, ALL mouse wheel.
• Don't worry about case - table, Table and TABLE are all the same.
• Substrings are searched - so use localiz instead of localize or localization, which will find both occurrences of the word.
• Use quotes to search for exact words or phrases

See the Bugzilla QuickSearch page for more tips.

You'll notice at the bottom of the search results there's an option to "Remember search as":

This is very handy, especially if you've taken the time to cobble together an advanced search. Saved searches are then set up as links in the page footer. In the image above you can see I currently have 2 saved searches: "My Bugs" and "My CC'd Bugs".

Once you're reasonably sure the bug you've found hasn't been filed yet, click the File a Bug button on the home page. You'll be prompted to select a classification, for which you want to choose Distribution.

This will take you to the Enter Bug form:

Select the Component that is most closely related to your problem. As you select the components you'll see their description in the text box to the right. If you are still unsure, you can look at similar bugs you found during your search. Don't worry if it's mis-classified - it will get corrected.

After selecting the Version of OpenSolaris you are using, select the Severity. Although all bugs may feel like mission critical blockers to you, please try to be realistic with this setting:

Blocker	Blocks development and/or testing work
Critical	crashes, loss of data, severe memory leak
Major	major loss of function
Normal	regular issue, some loss of functionality under specific circumstances
Minor	minor loss of function, or other problem where easy workaround is present
Trivial	cosmetic problem like misspelled words or misaligned text
Enhancement	Request for enhancement

Note the Enhancement value. I use this one a lot to request features I'd like to see in the product.

The two most important fields are the Summary and the Description. A good summary will quickly and uniquely identify the bug. Provide as much detail in the description as you can, ideally providing the exact steps on how to reproduce the bug and any possible work-arounds you may have discovered (documented work-arounds are one of the great benefits of searching through the bug database). Finally, don't clutter the Description field with large amounts of output or log text. Rather, use the attachment button to attach the text or log. Likewise, if you can take a snapshot of the problem, attach that as well (there's a screen shot tool in Applications > Graphics > Save Screenshot).

See the Bug Writing Guidelines for more information.

When satisfied, press Commit to submit your bug. And thank you, you've just made a valuable contribution to OpenSolaris.

Read more >>

A little gem included with the Crossbow Virtual Wire Demo Tool is a DTrace script and associated Chime display written by Crossbow engineer Kais Belgaied. I've augmented the script a bit by adding an END clause to format the output, which shows the number of bits transferred over the period of time the script is run:

bleonard@opensolaris:~$ dtrace -s vWireBuilder/VWire/so.d 
Tracing... Hit Ctrl-C to end.
^C
Process              Zone       Protocol   Destination      Port           Bits

pidgin               global     TCP        202.43.216.31    5050             296
firefox-bin          global     TCP        66.196.87.126     443           13968

Trace ran for 29 seconds

The associated Chime display is useful when you want to monitor a system with lots of network traffic. In this example I'm using Netperf to simulate traffic between my global zone and a local zone, which has been configured to run at IP address 10.0.1.50:

In this example you can see we're achieving a throughput of 932,708,352 bits per second, or approximately 933 megabits per second. In my case the local zone is configured using a Crossbow VNIC, myzone0:

bleonard@opensolaris:~$ dladm show-vnic
LINK         OVER         SPEED  MACADDRESS           MACADDRTYPE         VID
myzone0      e1000g0      1000   2:8:20:59:0:b5       random              0

Notice the link speed of myzone0 is 1000 megabits per second, and at the moment it's using just about every bit of that bandwitdh, potentially starving other virtual NICs and the physical NIC itself from the network. Fortunately, Crossbow allows me to put some reigns on my virtual NICs. For example, if I do the following:

pfexec dladm set-linkprop -p maxbw=200 myzone0

Above we can see we're now running just under 200 megabits per second. Or More ridiculously:

pfexec dladm set-linkprop -p maxbw=2 myzone0

We're now under 2 megabits per second.

Now it is also possible to drill down on the Chime display, graphing throughput over time. The above scenario looks like so:

If you want to check this out on your own environment, you only need the D script, Chime and the Network display:

• The DTrace script: so.d
• Chime
  
• The Chime display: Network.xml

Note, you'll have to edit the programFileName property of Network.xml to the location you saved the D script:

 <void property="programFileName">
   <string>/export/home/bleonard/so.d</string>
 </void>

To launch the Chime display:

chime -C ./Network.xml

Read more >>

At the CommunityOne West keynote last June a demo of Crossbow was given using a tool called Virtual Wire. If you haven't seen the demo before, you can watch a 10 minute replay of it here:

Since that time a lot of folks have asked for access to the tool that was used in the demo. Browsing the Crossbow project page recently I noticed the Crossbow Virtual Wire Demo Tool has now been made available.

I'll warn you in advance that I view the tool as alpha quality as it doesn't have some basic functionalities like undo. However, as the names implies, it's perfect for demos (and learning a lot about zones and networking).

When you extract the tool you'll find a couple of screencasts, which I've also made available here. There's no audio, but they do give you the basic idea of the capabilities of the tool.

Demo 1: Creating and Connecting 2 Private Networks

This demo uses the Crossbow Virtual Wire Demo Tool to create 2 private networks. One network contains a web server (192.168.2) while the client is connecting from a separate network (193.168.3). A router is set up to forward packets between the 2 networks. This is a self-contained private network. The demo concludes by using a network load generator along with DTrace to show how easy it is to monitor and configure bandwidth to the server. The details on how to reproduce this demo have been documented here.

Demo 2: Creating a Private Network Connected to the Public Network

This demo uses the Crossbow Virtual Wire Demo Tool to create a private network containing a web server behind a firewall. The IP routing is configured to access the server from the global zone. The demo concludes by using a network load generator along with DTrace to show how easy it is to monitor and configure bandwidth to the server. The details on how to reproduce this demo have been documented here.

Read more >>

I've read that the best way to learn DTrace is to study existing D scripts. However, upon doing so I quickly find myself asking "where is that documented?" And it turns out, with tens of thousands of probes, as is often the case, it's not. So I'd love to hear what others with more DTrace experience do, but these are the steps I seem to have stumbled upon.

As an example, let's take the script /opt/DTT/Proc/rwbytype.d, which shows the bytes read and written by process by vnode type. Here's an example of the output:

bleonard@opensolaris:/opt/DTT/Proc$ rwbytype.d 
Tracing... Hit Ctrl-C to end.
^C
PID    CMD               VTYPE  DIR     BYTES
21124  gnome-terminal      chr    W         1
10523  gnome-power-mana   sock    W         4
10565  gnome-netstatus-   fifo    R         7
10547  gnome-netstatus-   fifo    W         8
4817   conky              fifo    R         9
10634  firefox-bin        fifo    R         9
10634  firefox-bin        fifo    W         9
15379  whoami             fifo    W         9
10547  gnome-netstatus-   fifo    R        20
10565  gnome-netstatus-   fifo    W        20
15377  dtrace              chr    W        31
10412  Xorg                chr    R        32
10469  gnome-settings-d   sock    R        32
10523  gnome-power-mana   sock    R        32
21124  gnome-terminal      chr    R        33
10472  metacity           sock    W        56
10472  metacity           sock    R       200
21124  gnome-terminal     sock    R       220
15379  conky               reg    R       529
15379  ksh93               reg    R       529
4817   conky              sock    R      2720
10412  Xorg               sock    W      3204
4817   conky               reg    R      6696
21124  gnome-terminal     sock    W      6852
4817   conky              sock    W     37952
10412  Xorg               sock    R     44864

And here's the D script that produces that output:

typedef struct vtype2str {
	string code;
};

translator struct vtype2str < int T > {
	/* the order has been picked for performance reasons */
	code =
	    T == 1 ? "reg"   :
	    T == 9 ? "sock"  :
	    T == 4 ? "chr"   :
	    T == 6 ? "fifo"  :
	    T == 8 ? "proc"  :
	    T == 2 ? "dir"   :
	    T == 3 ? "blk"   :
	    T == 5 ? "lnk"   :
	    T == 7 ? "door"  :
	    T == 10 ? "port" :
	    T == 11 ? "bad"  : "non";
};

dtrace:::BEGIN
{
	printf("Tracing... Hit Ctrl-C to end.\n");
}

fbt::fop_read:entry,
fbt::fop_write:entry
{
	self->type = xlate <struct vtype2str *>(args[0]->v_type)->code;
	self->size = args[1]->uio_resid;
	self->uiop = args[1];
}

fbt::fop_read:return
/self->uiop/
{
	this->resid = self->uiop->uio_resid;
	@bytes[pid, execname, self->type, "R"] = sum(self->size - this->resid);
	self->type = 0;
	self->size = 0;
	self->uiop = 0;
}

/* this is delibrately redundant code for performance reasons */
fbt::fop_write:return
/self->uiop/
{
	this->resid = self->uiop->uio_resid;
	@bytes[pid, execname, self->type, "W"] = sum(self->size - this->resid);
	self->type = 0;
	self->size = 0;
	self->uiop = 0;
}

dtrace:::END
{
	printf("%-6s %-16s %6s %4s %9s\n",
	    "PID", "CMD", "VTYPE", "DIR", "BYTES");
	printa("%-6d %-16s %6s %4s %@9d\n", @bytes);
}

First of all, this D script is only enabling 6 probes:

dtrace:::BEGIN
dtrace:::END
dtrace::fop_read:entry
dtrace::fob_read:return
dtrace::fob_write:entry
dtrace::fob_write:return

The BEGIN and END clauses are self-explanatory. Where I get tripped up is looking at the 2nd clause:

fbt::fop_read:entry,
fbt::fop_write:entry
{
	self->type = xlate <struct vtype2str *>(args[0]->v_type)->code;
	self->size = args[1]->uio_resid;
	self->uiop = args[1];
}

First, I see that we're setting 3 thread-local variables, type, size and uiop. All of these variables are set using probe arguments, and this is where things start to get confusing - where are these arguments defined? I can run the following dtrace command to get verbose information on a probe:

bleonard@opensolaris:/opt/DTT/Proc$ dtrace -lv -n fbt::fop_read:entry
   ID   PROVIDER            MODULE                          FUNCTION NAME
17725        fbt           genunix                          fop_read entry

	Probe Description Attributes
		Identifier Names: Private
		Data Semantics:   Private
		Dependency Class: Unknown

	Argument Attributes
		Identifier Names: Private
		Data Semantics:   Private
		Dependency Class: ISA

	Argument Types
		args[0]: vnode_t *
		args[1]: uio_t *
		args[2]: int
		args[3]: cred_t *
		args[4]: caller_context_t *

So I learn that the data semantics of the argument attributes are private, which means this probe is accessing private implementation details of the kernel, but that's the case with most DTrace kernel probes. Of more interest is the argument list:

args[0]: vnode_t *
args[1]: uio_t *
args[2]: int
args[3]: cred_t *
args[4]: caller_context_t 

So I can see there are 5 total arguments available for this probe and I know my clause is accessing args[0] and args[1]. But what now? What exactly is vnode_t and uio_t? As far as I know, there's no further dtrace command to drill down into these data types. At this point I turn to the OpenSolaris Source Browser.

I begin by searching on the probe function fop_read, which returns 2 results, the vnode.h header file and the vnode.c source file:

Searched refs:fop_read (Results 1 -2 of 2) sorted by null

/onnv/onnv-gate/usr/src/uts/common/sys/
vnode.h	[all...]
/onnv/onnv-gate/usr/src/uts/common/fs/
vnode.c	3177 fop_read( function     [all...]

The fop_read function is defined on line 3177 of vnode.c and I can click on that link to be taken directly to the source. Sure enough, the arguments to the fop_read function in vnode.c match what I was told by dtrace:

   3176 int
fop_read(
	vnode_t *vp,
	uio_t *uiop,
	int ioflag,
	cred_t *cr,
	caller_context_t *ct)

Now I can click on vnode_t to see its definition and this returns 3 results - the logical one being the vnode.h header file returned by our first search:

/onnv/onnv-gate/usr/src/uts/common/sys/ vnode.h 257 } vnode_t; typedef in typeref:struct:vnode 800 int (*vop_open)(vnode_t **, int, cred_t *, \ 802 int (*vop_close)(vnode_t *, int, int, offset_t, cred_t *, \ 804 int (*vop_read)(vnode_t *, uio_t *, int, cred_t *, \ 806 int (*vop_write)(vnode_t *, uio_t *, int, cred_t *, \ 808 int (*vop_ioctl)(vnode_t *, int, intptr_t, int, cred_t *, \ 810 int (*vop_setfl)(vnode_t *, int, int, cred_t *, \ 812 int (*vop_getattr)(vnode_t *, vattr_t *, int, cred_t *, \ 814 int (*vop_setattr)(vnode_t *, vattr_t *, int, cred_t *, \ 816 int (*vop_access)(vnode_t *, int, int, cred_t *,     [all...]

Here we see that vnode_t is defined on line 257, which you can click to navigate directly to that section of the source. Our D script above is referencing args[0]->v_type, and we can see v_type is an enum:

typedef struct vnode {
	kmutex_t	v_lock;		/* protects vnode fields */
	uint_t		v_flag;		/* vnode flags (see below) */
	uint_t		v_count;	/* reference count */
	void		*v_data;	/* private data for fs */
	struct vfs	*v_vfsp;	/* ptr to containing VFS */
	struct stdata	*v_stream;	/* associated stream */
	enum vtype	v_type;		/* vnode type */
	dev_t		v_rdev;		/* device (VCHR, VBLK) */

Which you can also click on to see the type definition:

/*
 * vnode types.  VNON means no type.  These values are unrelated to
 * values in on-disk inodes.
 */
typedef enum vtype {
	VNON	= 0,
	VREG	= 1,
	VDIR	= 2,
	VBLK	= 3,
	VCHR	= 4,
	VLNK	= 5,
	VFIFO	= 6,
	VDOOR	= 7,
	VPROC	= 8,
	VSOCK	= 9,
	VPORT	= 10,
	VBAD	= 11
} vtype_t;

Which matches up perfectly to the struct at the beginning of our D script. So, now I've got a better handle of exactly what's going on with this D script.

Now, as a test, see if you can follow the same procedure to find the documented comment in the OpenSolaris source code for arg[1]->uio_resid :-). 

Read more >>

Sun ISV engineer and DTrace evangelist Angelo Rajadurai has just authored a new quick start guide to DTrace. The guide is based on his "Observing Applications with DTrace" presentation (slides, video) at CommunityOne West and focuses on tracing web applications. The examples from the book have also been added to a wiki page for easier copy & paste.

Read more >>

A hidden little gem that comes included with your installation of OpenSolaris is the DTrace Toolkit. You'll find it installed at /opt/DTT.

The DTrace Toolkit is a collection of tools written by volunteers of the DTrace community, led by Brendan Gregg. What I find most impressive is how well it has been put together. There are 230 executable DTrace scripts, each with their own man page. The scripts are organized by type (CPU analysis, I/O analysis, etc.) so they are easy to study, but they've all been symlinked into a Bin directory, so they are easy to put on your path. I've added the following to my ~/.bashrc file:

#
# DTrace Toolkit
#
PATH=$PATH:/opt/DTT/Bin
MANPATH=$MANPATH:/opt/DTT/Man

In addition to their man pages, all of the scrips have a -h help option, for example:

bleonard@os200906:~$ execsnoop -h
USAGE: execsnoop [-a|-A|-ehjsvZ] [-c command]
       execsnoop                # default output
                -a              # print all data
                -A              # dump all data, space delimited
                -e              # safe output, parseable
                -j              # print project ID
                -s              # print start time, us
                -v              # print start time, string
                -Z              # print zonename
                -c command      # command name to snoop
  eg,
        execsnoop -v            # human readable timestamps
        execsnoop -Z		# print zonename
        execsnoop -c ls         # snoop ls commands only

Note, if you want to run the DTrace scripts from your user account, run the following command to add the needed privileges, substituting in your user name:

pfexec usermod -K defaultpriv=basic,dtrace_user,dtrace_proc,dtrace_kernel <user>

To get started learning all that's available, I recommend reading the DTrace Tookkit page. Also, Stefan Parvu has an excellent presentation on DTrace and the toolkit that's worth a look.

With this toolkit there's probably little need to write a DTrace script from scratch - it's literally a treasure chest of material that I've only begun to explore.

Read more >>

There are (at least) 2 big Linux conferences this weekend, the Ontario Linux Fest and the Florida Linux Show. The good news is that you'll find OpenSolaris represented at both of them. In Ontario Canada, Toronto OpenSolaris User Group leader Steven Acres will deliver Why OpenSolaris? In Orlando, Florida, I'll be delivering What's So Cool About OpenSoalaris Anyhow?

If you happen to be nearby (or willing to travel), please stop by and say hello.

Read more >>

When I edit a text file in vi, I find it much easier to read if the syntax is color highlighted. There are 2 steps to enabling this. First, add an alias for vi (or vim) to your ~/.bashrc file as follows:

#
# Enable color support for vi
#
alias vi='TERM=xtermc vim'

You can read the gory details about why this needs to be done in bug 1641 - xterm terminfo entry doesn't support color.

You also need to tell vi to use syntax coloring. Adding the following to your ~/.vimrc file (create one if it doesn't yet exist):

syntax on

Much nicer:

Read more >>