NetBSD Code-In 2012/2013 wiki

The developers of NetBSD have made the current development sources available to the public for several reasons. Overall, providing NetBSD-current helps us to create a more stable, accessible system.

It makes it easier for people to become involved in the development of NetBSD. Distributing the current development sources allows a greater number of people to see where the system is going, and to become involved with new features as they are implemented.

It also makes changes from users easier to integrate. If users make changes against the current development sources, then virtually no integration is needed to get them into the master source tree.

It also allows wider testing of the software as it is developed. Users of NetBSD-current are encouraged to send in bug reports about the current sources, and that helps find and fix bugs. Because people are testing the software soon after it's written, more bugs can be found and eliminated.

• People using NetBSD-current are strongly encouraged to subscribe to the current-users mailing list. The source-changes mailing list is also of interest.
• When upgrading to a more recent version of -current you should always install and boot a new kernel before installing any new libs (*). In general the best approach is to try the new kernel before anything else, and if you hit any problems see the entry in the Kernel FAQ.
• When compiling a -current kernel, always remember to include the COMPAT _ <lastrelease> option (e.g., COMPAT _ 16). As current diverges from the last s”table release, compatibility code will be added, but it will only be enabled if this option is present. At a bare minimum, you will need this compatibility code for the time between booting the new kernel and finishing your build via build.sh

\*: Unless you are certain there have been no new system calls added, but do it anyway; it's safer.

Please remember to check src/UPDATING for quirks around certain specific changes.

To quickly begin using current, start with a snapshot generated by release engineering. The current status of each platform can be seen at NetBSD Autobuild and the corresponding releases found in by date and platform.

1. Hunt down to the desired binary/sets directory, and mget \*.tgz files into your favorite local administrative directory (for example, $HOME/current ); when limited by disk space and/or time, only kern-GENERIC, etc, base, and comp (if you want a compiler) are essential.
2. Extract the desired kernel (usually GENERIC), copy it into (root) directory.

$su
# cd /root
# tar -zxpf ~/kern-GENERIC.tgz
# ln -fh /netbsd /netbsd.old
# cp netbsd /netbsd.new
# ln -fh /netbsd.new /netbsd

Warning: Don't extract any userland binary sets before rebooting your machine with the new kernel. Newer binaries might use new system calls an old running kernel doesn't support.

1. Check if there are any other files which might also be required by a new kernel. Again, src/UPDATING might mention possible quirks on daily changes. The following items are typical files that possibly need to be updated:
   1. bootloader
      
      Usually a machine specific bootloader passes several parameters to a loaded kernel. If some new parameters have been added or some existing APIs between bootloader and kernel are changed you might also have to install new bootloader files for a new kernel to handle new features. A method to update bootloader files is quite machine dependent, so check boot(8) and installboot(8) man pages for details.
      
      On i386 and amd64, if you are using FFSv1 for root file system on wd0a (i.e. first ATA drive), typical commands to update bootloaders are:

# tar -C /tmp -zxf ~/base.tgz ./usr/mdec
# cp /tmp/usr/mdec/boot /
# installboot -v /dev/rwd0a /tmp/usr/mdec/bootxx_ffsv1

If you are using FFSv2 for root file system use the following commands instead:

# tar -C /tmp -zxf ~/base.tgz ./usr/mdec
# cp /tmp/usr/mdec/boot /
# installboot -v /dev/rwd0a /tmp/usr/mdec/bootxx_ffsv2

Note /usr/mdec/bootxx_ffsv1 and /usr/mdec/bootxx_ffsv2 are primary bootloaders which are file system dependent. /usr/mdec/boot is secondary loader and it's file system independenet.

If you forget your root file system type (FFSv1 or FFSv2), you can check it by dumpfs(8) command:

# dumpfs /dev/rwd0a | head -3
file system: /dev/rwd0a
format  FFSv2
endian  little-endian
#

b. kernel modules

A new framework “kernel modules” has been introduced after netbsd-5 was branched, and GENERIC kernel on i386 port has been switched to using the kernel module files since November 2008. The kernel module files will be loaded dynamically by the kernel to support various kernel options (including file systems) on demand, rather than linking all necessary (but possibly unused) object files into the kernel binary. This means if you are trying to boot a new GENERIC kernel, you also have to prepare new kernel module files for the new kernel.

To prepare new kernel module files, you can simply use a new modules set file which has been prepared since September 2009:

# cd /
# tar -zxpf ~/modules.tgz

Note i386 port also provides MONOLITHIC kernel binary in kern-MONOLITHIC.tgz set file since October 2009. The MONOLITHIC kernel includes all necessary options in its kernel as well as 5.0 and prior GENERIC kernels and it doesn't depend on kernel module files at all. If you would just like to test new features of a new kernel without updating kernel modules, using MONOLITHIC kernel is easier way for the first and quick trial.

It's also a good idea to put an old MONOLITHIC kernel into / (root) directory for emergency and recovery because if newer modules have some fatal issue there is no easy way to specify an alternative path of old module files to a modular'ized kernel (and you can't rename directories without a working kernel).

Warning: The infrastructure of kernel module files mentioned here is still under discussion on -current development. It could be changed at some point before the next 6.0 release and in that case the description in this section will be obsolete. Again, check src/UPDATING and current-users mailing list for updated information.

There is a possible alternative structure for kernel modules which was proposed on May 2009, but we have not got any conclusion yet. This would be because most -current users build their own custom kernels from sources, but kernel modules might be rather useful for users who don't want to bother to compile their own kernels from sources to just try to use optinal functions. Anyway, any feedback about this brandnew kernel modules is quite appreciated.

1. Reboot machine with the new kernel:

# shutdown -r now

5. Make sure the new kernel boots and works properly. If your new kernel has any trouble, you can recover it by loading the renamed old one. If you are using modular'ized GENERIC kernel mentioned above, you might also have to restore old kernel module files.

1. Extract the matching base, and any other desirable feature sets except etc:

$ su
# cd /
# tar -zxpf ~/base.tgz
# tar -zxpf ~/comp.tgz
# ...

Don't forget to specify “p” option (preserve permissions) on MAN.TAR.1 command otherwise setuid'ed commands (like MAN.SU.1) won't work.

Warning: Extracting etc.tgz on the installed system will overwrite your local settings.

1. Update /etc as last step: postinstall(8) will first check and fix most things that can be automated, and etcupdate(8) in the second step will ask on what to merge:

# /usr/sbin/postinstall -s ~/etc.tgz check
# /usr/sbin/postinstall -s ~/etc.tgz fix
# /usr/sbin/etcupdate -s ~/etc.tgz
# shutdown -r now

If you have the X sets installed (xbase, …), you can repeat the postinstall and etcupdate steps with xetc.tgz as argument before rebooting.

At this point, you are relatively current and ready to build your own current source.

See the Obtaining the sources section in the NetBSD Guide.

See the Crosscompiling NetBSD section in the NetBSD Guide.

See the Updating an existing system from sources section in the NetBSD Guide.

See the More details about the updating of configuration and startup files section in the NetBSD Guide.

If you try to build -current, either from a snapshot or an earlier -current, and it doesn't work, don't panic. Try these steps:

1. Read the src/UPDATING file from the release you're trying to build.
2. Read the current-users archive for hints.
3. Update again. You may have caught the repository in the middle of a commit to several related files, or the problem might have already been fixed.
4. If all else fails, send email to current-users explaining the problem. Include the date, time, and method you used to get your -current sources, as well as any local changes you've made. Then put in a short script that includes the error messages you're getting. Somebody will probably fix the problem momentarily.

See the Fetching by CVS section in the NetBSD Guide.

  $ cvs checkout -D 20020501-UTC src

  $ cvs checkout -rnetbsd-5-0 src

See src/doc/BRANCHES for a description of the branches in the CVS repository.

• Do not use the cvs '-z' flag. The data stream gets out of sync, leading to corruption on the client, or causing the client to hang completely. The additional load is also hard on the cvs server.
• If you want to check out a certain branch of the tree, you may want to take caution not to overwrite any existing directories by creating a new directory for this branch:

$ cd /parent/dir/to/checkout/into
$ mkdir NewName-temp
$ cd NewName-temp
$ cvs checkout ... src
$ mv src ../NewName
$ cd ..
$ rmdir NewName-temp

- You will have to use objdirs in order for cvs updates to work correctly. If you happen to get errors from cvs saying things like:

   cvs [update aborted]: could not chdir to gnu/usr.bin/gdb/gdb: Not a directory

you should do a make cleandir and try again. Make sure to run make obj after the cvs update.

• You can put switches for specific commands in a .cvsrc in your home directory, and they will be automatically used. A sample .cvsrc would be:

 update -dP
   checkout -P
   diff -u

Sources are imported as follows:

  $ cvs -d /misc/cvsrep import -I ! -I CVS netbsd netbsd current-

date is replaced by the date of the SUP for tracking purposes. The -I ! -I CVS options ensure that no file in the source tree is ignored except 'CVS' directories. This is because some NetBSD source files have extensions which are normally ignored by CVS. If there are any conflicts with local patches the import command will report them and will describe a command to merge the conflicts something like:

  $ cvs checkout -jnetbsd:yesterday -jnetbsd netbsd

This merge command will correctly merge the imported NetBSD sources but it will not handle the removal of files locally which have already been removed by the SUP process. To do this the merge command would be:

  $ cvs update -j -j current-

previous import tag should be replaced with the name of the tag used for the previous cvs import. date should be replaced with the current date to yield the same tag used on the current import that has just been merged.

The conflicts reported by the import command are potential conflicts. These are usually merged by the update command but in some cases a real conflict occurs. In these cases a manual merge of the conflicting lines will be required. A real conflict will be reported in the cvs update output as a C followed by a filename.

Merging conflicts manually is not a simple process but in most cases it should be resolved by removing the local changes and making the file like the original NetBSD source code.

CVS marks conflicts as follows:

  <<<<<<
  code from local file
  ======
  code from imported file
  >>>>>> local revision number of newly imported revision

If the import reports no conflicts the checked out copy of the tree should be updated in exactly the same way as for the conflicts case.

All update and checkout commands should be done in the directory where the sources have been checked out. On my system this is /usr/src/netbsd.

If this is the first import then there will be no sources checked out. Assuming you wish to create the source tree in '/usr/src/netbsd' The following commands will check out the source and no merge step is required.

  $ cd /usr/src
  $ cvs -d /misc/cvsrep checkout netbsd

If the build completes successfully, and produces a working set of binaries, it can be useful to tag the working sources. This allows rewinding to a working build tree with a single CVS command in the event that the current tree becomes unbuildable for any reason. This can be performed by issuing the following command:

  $ cvs tag successful-build-

• If the NetBSD customised version of CVS, which recognises \$Net BSD\$ markers in files, is not used, the NetBSD revision number of the file is available for reference purposes when build problems occur.
• The sup/import/merge sequence described above is quite easily automatable. The following Perl script automates this process.

#!/usr/pkg/bin/perl
#
# Script to SUP NetBSD-current, import it into CVS and merge it with
# any local changes.
#
# NOTES:
# This script does no error handling so is not really suitable for
# non-interactive use.
#
# This script has only been test with cvs-1.10.1 and cvs-1.9.18.
#
$SRCROOT="/usr/src/netbsd";
$IMPORTROOT="/misc/import";
$CVSROOT="/misc/cvsrep";
#run the sup into a perl stream
system "/usr/sbin/sup -zsv" ; # This may need to change for none
  # current systems
# now import the new files into CVS

chdir $IMPORTROOT or die "Could not cd to $IMPORTROOT\n";

($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime;
$date = localtime;
$shortdate = sprintf "%02d%02d%04d",$mday,$mon+1,1900+$year;
system "/usr/local/bin/cvs -d$CVSROOT import -I ! -m\"SUP Import $date\" netbsd netbsd current-$shortdate ";

# make the working directory the local NetBSD Tree
chdir $SRCROOT or die "Could not change to $SRCROOT directory\n";

# Now do the import.
$lastimport = `cat /usr/src/netbsd/.tag`; # `s are backquotes
$lastimport =~ s/\n//; # strip off any trailing newline in the string
system "/usr/local/bin/cvs update -j $lastimport  -j
current-$shortdate ";
# Now write the current file into tag save file
open TAG,">$SRCROOT/.tag" or die "Could not open new tag file";
 print TAG "current-$shortdate";
close TAG;

This script was written in Perl since it is the scripting tool which the author has the most experience with. It should be fairly straightforward to write a shell script to perform the same task.

• Techniques for tracking current with CVS have been discussed several times on the NetBSD current-users mailing list. For alternative techniques try searching the NetBSD mailing lists.

If you have any comments or suggestions please send them to Mike Pumford mpumford@black-star.demon.co.uk (who maintains this entry) or www@NetBSD.org.

All the procedures described above allow you keeping your own changes in your repository, which has its advantages if you develop your own software based on NetBSD. If you don't want to maintain your own CVS repository, but just want to mirror NetBSD's CVS repository, there are three ways to do so.

Each of the methods described briefly below will get you a copy of the NetBSD CVS repository (i.e. the RCS ,v files, not the checked out files!). You can then setup your own anoncvs server or check out to a local harddisk. It's also useful for fast access to the history information stored in the repository.

The methods to retrieve the whole repository are:

sup

• If you use sup already to mirror other parts of the NetBSD source, you will want to add the following lines to your sup config file:

anoncvs release=all  host=sup.NetBSD.org hostbase=/ftp/pub \
base=/usr prefix=/usr backup use-rel-suffix compress

After that, run “sup /path/to/supfile anoncvs” to retrieve the files.

Some example sup files are available in /usr/share/examples/supfiles. Also, check our list of SUP mirrors to find the server closest to you!

rsync

• Note that rsync puts quite a heavy load on our rsync server, and as such the number of concurrent rsync users is restricted. If you still want to try rsync, the command to retrieve the repository is:

rsync -v -a --delete --exclude '#cvs.lock' rsync://anoncvs.NetBSD.org/cvsroot/src .

Please see our list of rsync mirrors!

cvsup

• CVSup is not currently available for all NetBSD architectures, since the M3 compiler has not been ported. On i386, you can mirror the repository from cvsup.de.NetBSD.org with the devel/cvsup package and the following config file:

*default host=cvsup.de.NetBSD.org
*default base=/usr
*default prefix=/local/NetBSD-cvs
*default release=cvs
*default delete use-rel-suffix
*default compress

netbsd

Please see our list of CVSup mirrors!