Debugging of Sakai error messages
From TroubleshootingWiki
| Official Page |
| Project Documentation |
| Download |
|
The smooth running of a Sakai deployment and users' perception of its quality depends on how fast problems are dealt with. The sooner you deal with an issue the quicker it is resolved, and the less the disturbance to an end user. Being honest and admitting that occasionally there are errors does not imply that Sakai is buggy, just that it is designed from the bottom up to help system administrators to act fast.
Things can go wrong, a piston blown in a car engine, lack of petrol to get you to your destination, or without warning, a hanging office application. I have not interacted with software yet that is perfect. There is no strategy to provably remove every single defect from a given software product. The primitive computer used for the first lunar landings had as few as 10,000 lines of machine code to control it. The complexity of the task was high, but the number of lines that errors could hide in was low. A modern operating system contains tens of millions of lines of code. Sakai sits between these two extremes, with around half a million lines of active running code. To aggravate the potential for errors, each line of Java has the power to do a lot more work than one line of machine code. The use of a feature-rich language improves the programmer's productivity and enables the creation of more functionality in a shorter amount of time. However, even in a perfect universe with the most perfect quality assurance, there will always be some residual defects left. Under the pressure of high numbers of users, very occasionally an error will be triggered. Reducing pain and negative user perception depends on the administrator's ability to remove specific error types quickly, and for developers to find and patch the bug in the code base.
Even if perfect, the software still resides on a server that can go wrong. Servers have moving parts, such as the hard disk, that wear out; memory that solar radiations can hit; electrical supplies and CPUs that generate heat, fans that rotate. For small-scale deployments, you can limit risks with regular backups of the most important files, redundant hardware, and uninterruptible power supplies (UPS). A UPS supplies electricity when the power grid is disrupted for a long enough time to safely turn off the servers. For medium- to large-scale deployments, load balancers ensure a lack of dependency on any application server. If one application server fails, the load balancer spots this and then stops passing on requests to the failed server (this is known as failover). However, often the database is still a single point of failure and a cause of concern.
Another type of real-time exception is the consumption of all the available resources. On the busiest day of the year, as much as 10% of your registered Sakai membership is actively online. For example, if your organization has 160,000 students with accounts in Sakai, as many as 16,000 users may be logged on simultaneously. With peak user demand an order of magnitude higher than on a more normal quiet day, the server may consume all the available memory, hard disks may fill up, or network bottlenecks may emerge.
Luckily, Java comes to the rescue. Java is helpful in dealing with real-time problems, allowing the programmer in most situations to decide what to do next. The language uses exception handling. When an issue occurs, the running code passes control to another piece of code, which then deals with the exceptional situation or ignores it. Thus, unless the JVM has no choice, for example because there is no more memory, an application can deal with the problems signaled and keep on running.
This advanced tutorial will explain how to configure the logging and error reporting to provide focused early signals. The tutorial will also mention how analyzing error reporting helps debugging and long-term trend watching. A number of common error messages are also defined.
| Sakai is heavily dependent on Java, therefore, to understand the most common errors requires a basic understanding of this programming language. |
Contents |
A policy of containment of errors
Sakai is a complex application built out of hundreds of thousands lines of code. It is inevitable that even with the most vigerous testing a few residual bugs will occasionally cause problems for a small minority of end users. This section describes how these issues are found, contained, and reported back.
Reporting
Sakai has a policy of containment and signaling of issues. When all else has failed and there is no way that the application can usefully fulfill the end user's request, then a error form is returned to the end user.
At the top of the form is a space for user comments and at the bottom is the rather ugly and unreadable stack trace that the underlying code passes on. When an exception is thrown, it starts in a particular method as part of a Java class. The method might have been called by another method, which in its own turn may also have been called from within yet another method. Note that the depth of the stack trace, especially when you take into account the server interactions, makes for a very long list.
To the average human, stack traces are unreadable and give the feeling that they have trodden in something rather smelly. Even if the stack trace is stating an innocent fact, unless the user understands Java, he or she is going to feel that the system is shouting out very loudly that there are problems. Luckily, most stack trace types are dealt with early by the developer or during the Quality Assurance cycle before the offending code goes into production.
Once the user submits the report, Sakai sends the report via email to a specific address for collection and later analysis. You can set the target email address, like most other global configuration, in sakai/sakai.properties via defining the portal.error.email property.
The email that is sent out has a fixed structure that makes it possible to process the text relatively easily via scripts; see the following example listing:
from Sakai QA Network qa1-nl (svn tags/sakai_2-5-3_rc03) using MySQL 5/InnoDB, Java 1.5.0_13 amd64 <no-reply@qa1-nl.sakaiproject.org> to xxxx@xxx.xx date 13 October 2008 13:32 subject Bug Report: F101671D49BBA3E4A1A59338257B074CA307FB52 / null user: null (null) email: null usage-session: null stack-trace-digest: F101671D49BBA3E4A1A59338257B074CA307FB52 sakai-version: 2-5-3 service-version: app-server: qa1-nl request-path: /portal/tool/!gateway-410 time: Oct 13, 2008 14:32:01
stack trace:--Long stack trace removed for readability
The email's from field includes a brief summary of the server and its capabilities. There are extra email fields containing the user name, time, email address of the user, and the request-path. The request-path is the web address that triggered the offending piece of code's error. In the listings case, the !gateway-410 part is the name that Sakai recognizes for a specific tool. The request-path not only indicates the rough location of the error in the code base, but also is enough for the code maintainers to work out who needs to write a patch to solve the specific issue. The error handling supplies the stack trace for later analysis. The associated stack trace digest is a unique hexadecimal number that is the same for two duplicate stack traces. This number allows you to keep track of the number of unique error types and to count the specific bug's relative importance.
Quality Assurance analysis
There are over ten servers scattered across the world, on which volunteers test functionality. As new organizations become involved with Sakai and they begin to invest time and resources into the community effort, occasionally they add and then maintain a new QA server to the list. For example, in late 2008 the community welcomed a server in Japan and another in Australia. This has the advantage of giving the specific donor organization insight into the current issues running with the newest versions of Sakai and a playground to train system administrators and generally build real-world operational experience.
When an error is generated on one of these QA servers, mail is sent to a specific mailbox. A script running on the Amsterdam QA server checks the mailbox every five minutes and, if a new email arrives, then report generation is immediately triggered and a related web page is updated with a detailed summary of ongoing errors.
The report includes a summary of all error forms sent and details of the stack traces.
To aid the debugging process, as part of the report, another script checks the source code for known bug patterns (http://qa1-nl.sakaiproject.org/codereview/bug_dashboard/detail_stack.html) and a link to those issues is made within the detailed report for each part of the stack trace, as shown in the following screen grab.
By clicking on the class name in the stack trace column, such as org.sakaiproject.site.tool.SiteAction, a detailed report of potential bugs automatically found for the class appears in your web browser. Clicking on the line number next to the class name returns an HTML version of the source code, with the focus on the correct line number.
This automatic analysis of bugs found during the QA process can speed up code cleaning during development.
Production systems
The University of Cape Town (https://vula.uct.ac.za) is an early adopter of the most up-to-date version of Sakai. The administrators keep track of the sent error forms and the impact of particular errors on the end user experience based on the purpose of the tool. For example, the negative impact of losing your place in a test is higher than your disruption during a search for documents. Intermittent issues due to server load have the potential to generate many low-level user irritations. Reading error reports gives you an understanding of the scale of irritation levels.
The report makes it very clear where any long-term issues reside and provides an indication for system administrators of where they should focus their efforts first. By comparing reports from different Sakai versions, you can see where you have successfully cleaned up issues and where new issues are emerging. Early adopters work closely with the community to iron out the residual defects so the rest of us do not have to.
Configuring logging
When you start up the demonstration, logging goes to your command console and a lot of text flashes past your eyes. The motivation for this start up verbosity is that if you have any misconfiguration, you will see their warnings early.
Within Sakai, the two most commonly used logging frameworks are Apache common logging (http://commons.apache.org/logging/) and Log4j (http://logging.apache.org/log4j). Despite the use of more than one framework, a Log4j configuration file manages all the logging within Sakai and allows for central control of all aspects of the logging. You can find code the relevant Log4j property file within the source at /kernel/log-configure/src/conf/log4j.properties. Any changes made to this file are only seen the next time Sakai starts up.
Log4j is a popular Java framework for logging application events. It has a reputation for efficiency. The framework separates the formatting and the destination of logging from the programming of the logging code. There is also a facility for sending log messages to the console, files, database, over the network through email or Telnet, and so on. The framework is so successful that it has been translated to work with Perl, PHP, C, and other programming languages.
Logging allows you to write messages at different levels of priority. The least significant message is trace followed by debug, info, warn,error, and for the worst conditions possibly fatal. It is therefore important for administrators that the application log consistently and to the right level of priority, and and that it use logging throughout the code base.
Log4j performs, so the developers can risk verbose logging, as long as it is fired off at the right level, and leave the configuration file to decide when to print the message or not. Log4j can filter messages based on log level and can have different log levels for different parts of the code. For example, it may be more important to have extra logging for potential database issues rather than for a particular tool.
Simply adding the following line to the global Log4j properties file will achieve this goal:
log4j.logger.org.sakaiproject.component.framework.sql.BasicSqlService=DEBUG
However, when you start your demonstration instance again, you will end up seeing a vast amount of extra logging similar to this:
DEBUG: Delete 0: delete from SAM_PUBLISHEDITEM_T where ITEMID=? (2009-01-26 12:37:22,130 main_org.hibernate.persister.entity.AbstractEntityPersister)
The line part log4j.logger tells the logger that this is a filter. The rest of the line is the name of the part of Java (known as a package) you want to filter against followed by an equal sign and then the logging level.
Examining the default log4j.properties file, the default logging level is warn. However, a number of packages have their level set to INFO. The MyFaces framework is too verbose and has had its logging level set higher to error.
You can also configure the filtering from within sakai/sakai.properties in a slightly different format. To set BasicSqlService to DEBUGyou will need to add the following two lines:
log.config.count=1 log.config.1=DEBUG.org.sakaiproject.component.framework.sql.BasicSqlService If you want to configure more than one filter, the count number will have to increase, for example: log.config.count=2 log.config.1=DEBUG.org.sakaiproject.component.framework.sql.BasicSqlService log.config.2=ERROR.org.sakaiproject.portal
The demonstration logging uses an appender to stdout, which is the console. This can be seen by the line: log4j.appender.Sakai=org.apache.log4j.ConsoleAppender
The appender decides the structure of the sent message using a conversion pattern defined via: log4j.appender.Sakai.layout.ConversionPattern=%p: %m (%d %t_%c)%n
The rules for the conversion patterns are fully described at http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/PatternLayout.html.
I prefer the timestamp at the beginning of the log entry and with as much detail as possible. The following line reconfigures Sakai to do this:
log4j.appender.Sakai.layout.ConversionPattern=%d{ABSOLUTE} %5p %c{1}:%L - %m%n
And a typical log entry will now look similar to: 12:46:07,767 INFO ContextLoader:197 - Root WebApplicationContext:
initialization completed in 9 ms
To change from sending to stdout to a log file that is rotated once a day is as simple as adding the four following lines:
log4j.appender.Sakai=org.apache.log4j.DailyRollingFileAppender log4j.appender.Sakai.DatePattern='.'yyyy-MM-dd log4j.appender.Sakai.File=/usr/local/tomcat/logs/tomcat.log log4j.appender.Sakai.layout=org.apache.log4j.PatternLayout log4j.appender.Sakai.layout.ConversionPattern=%p %d %t_%c%n%m%n
Notice the location of the log files is under /usr/local/tomcat/logs and the files start with tomcat.log followed by the daily time stamp.
Another useful place to append to is a Syslog daemon. Syslog (http://www.faqs.org/rfcs/rfc3164.html) is the logging protocol standard on most Unix/Linux boxes. A log server listens for log messages and the server can be configured to print the messages out to the local file system and/or send the messages on to other Syslog daemons for backing up or centralization. To append to the Syslog daemon, add the following five lines of extra configuration:
log4j.appender.SYSLOG=org.apache.log4j.net.SyslogAppender log4j.appender.SYSLOG.syslogHost=xxxx.xxxx.xxxx.xxxx log4j.appender.SYSLOG.layout=org.apache.log4j.PatternLayout log4j.appender.SYSLOG.layout.ConversionPattern=%p: %c - %m log4j.appender.SYSLOG.Facility=USER
Where xxxx.xxxx.xxxx.xxxx is the IP address of the Syslog daemon.
In summary, using one central configuration file, it is possible to modify the logging location, what is logged or ignored for logging, and the structure of the log messages.
Common error messages
There are numerous ways that Sakai, as a web-based application, can fail, mostly leaving log file evidence that is ready for interpretation behind.
This section's purpose is to give you a glimpse of which types of issue can exist. It explores a small set of messages. The methodology that was used for discovery purposes was to simulate issues by deliberately breaking the demonstration in a number of interesting ways.
During the experiments, failures forced the consumption of CPU time causing the test computer to heat up, generated many gigabytes of log file within a few seconds, and generally harried and stressed the server.
Java version
Maven compiles Sakai 2.5 or above with version 1.5 of Java. If you run the application with an old version of Java such as version 1.4, the JVM will generate the following error message:
Unsupported major.minor version 49.0
49 is the version number Sun uses to represent Java 1.5!
Java is backwardly compatible. Therefore, if your system has a higher version of Java deployed, such as version 1.6, then no problems will ensue.
Production servers sometimes have more than one version of Java installed. To ensure that the startup script can see the correct version, verify that JAVA_HOME is pointing to the top level of the Java installation, and that the PATH variable includes a link to the bin directory underneath.
Port issues
To allow more than one application to communicate at any one time on a given server, a specific application can assign itself a port for an associated IP address. There are around 65,000 ports with common port numbers below 1024 reserved for well-known protocols such as POP, SMTP, and HTTP.
When you start up Sakai, you may see the following error:
java.net.BindException: Address already in use
This is stating that the port number that Sakai tried to reserve is already being used. This can happen if you have another Java-based application like uPortal that uses the same port numbers for demonstrations. It can also happen when you have a proxy server on the same machine, which also uses this port by default.
To change the port number that Sakai wants, simply edit the Tomcat configuration found under conf/server.properties and modify port=8080 to the relevant number:
<Connector port="8080" maxHttpHeaderSize="8192" URIEncoding="UTF-8" maxThreads="150" minSpareThreads="25" maxSpareThreads="75" enableLookups="false" redirectPort="8443" acceptCount="100" connectionTimeout="20000" disableUploadTimeout="true" />
Tomcat consumes other ports as well. It listens via the Telnet protocol for a shutdown command, configured via the server.xml stanza:
<Server port="8005" shutdown="SHUTDOWN">
Tomcat talks with the Apache server via a custom protocol. Within server.xml, you can change the port number by modifying the line:
<Connector port="8009" enableLookups="false" redirectPort="8443" protocol="AJP/1.3" />
If you are trying to run two different demonstration versions of Sakai instances on the same machine, you will also need to edit the AJP shutdown port locations. Sakai itself may also hold open the default email port 25, that is if the service is enabled in sakai/sakai.properties.
Another point to note is that if Sakai wishes to run on a port below 1024 on a Unix/Linux system, then it needs to be run under a root user's account. Otherwise, you will see an error message similar to:
java.net.BindException: Permission denied:80 at org.apache.tomcat.util.net.PoolTcpEndpoint.initEndpoint (PoolTcpEndpoint.java:298)
Notice that the number 80 is the port number, which is default for a web server.
Out of memory
Modifying the startup script start-sakai.sh (or .bat) to starve the demonstration of memory is straightforward. Let us do this for the main heap space, where Java objects are kept during their life, and another memory area known as permspace.
The main task of permspace is to store information, a kind of design plan, about Java classes so that that the JVM can create objects later from the plan. Sakai uses numerous frameworks and third-party libraries and each web application has its own space. The more libraries, the larger the range of Java classes used. This reuse naturally translates into a large use of permspace. Permspace also stores string constants from the running application as well.
One of Java's advanced features is that of reflection. A Java program can look at a live object and work out its structure, such as which methods exist. Frameworks can also use reflection to construct and instance an object programmatically. For example, this is useful for when information about a Java class is stored in XML and then a framework wants to create a real instance of that object for use in a running application. Spring and Hibernate are good examples of this genre. The JVM also keeps the housework for reflection in permspace. For Sakai, it can be a crowded space, simply due to the sheer volume of functionality and compiled code, made even more crowded by reflection-driven frameworks.
To limit permspace to 16 MB, instead of the default setting of 256 MB requires changing the JAVA_OPTS and CATALINA_OPTS value -XX:MaxPermSize from 256m to 16m, for example, -XX:MaxPermSize=16m
Starting up Sakai generates the following error message:
15:36:15,481 INFO XmlBeanDefinitionReader:293 - Loading XML bean definitions from file [/home/alan/Sakai/instances/sakai-2-5-x/components/sakai-chat-pack/WEB-INF/components.xml] java.lang.reflect.InvocationTargetException Exception in thread "main" java.lang.OutOfMemoryError: PermGen space Exception in thread "Thread-0" java.lang.OutOfMemoryError: PermGen space
The message is clear: we have run out PermGen memory. The exact cause has to do with reflection. The end result is that the server stops and there are a few lines in a log file explaining clearly what went wrong. However, in the heat of battle the log file may well be full of messages and the one you need to understand be hidden in the forest of words. The simplest solution for a system administrator, if on a Unix/Linux box, is to search the log file for OutOfMemoryError with a command line tool such as grep.
To provoke the same reaction for the main heap space where live objects are kept, change the setting -Xmx1024m down to 64m. On restarting, the outputted error message includes:
Exception in thread "Thread-0" java.lang.OutOfMemoryError: Java heap space
On my local test machine, decreasing heap space down 16 MB instead of 1024 MB provoked a second well known type of response from my Sakai instance: it slowed to a stop and became unresponsive. This is worse than a failure, and without either the use of JMX or setting JVM logging settings is difficult to diagnose. The JVM is spending all its time garbage collecting and has no time to run the application. It is thrashing the CPU. To make matters worse, trying to stop the application via stop-sakai.sh (or .bat) turns out to be impossible and returns an error message similar to:
Using CATALINA_BASE: /home/alan/Sakai/instances/sakai-2-5-x Using CATALINA_HOME: /home/alan/Sakai/instances/sakai-2-5-x Using CATALINA_TMPDIR: /home/alan/Sakai/instances/sakai-2-5-x/temp Using JRE_HOME: /usr/lib/jvm/java-1.5.0-sun-1.5.0.16 14:38:54,199 ERROR Catalina:404 - Catalina.stop: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
You are left with only more radical ways to stop the process. In Unix/Linux, you can find the process ID of the runaway JVM via:
ps -ef alan 17232 1 23 14:52 pts/0 00:00:45 /usr/lib/jvm/java-1.5.0-sun-1.5.0.16/bin/java -server -Xmx1024m -XX:MaxNewSize=256m -XX:MaxPermSize=256m -Dsakai.demo=true
And when killing the process, the -9 signal tells the process to stop immediately. The 17232 is the process ID.kill -9 17232
If you are running on a Windows box, the tried and tested approach is to destroy the process from the task manager via the Ctrl-Alt-Delete key combination.
If you consider periodic demand, then under high load, Java web applications have the potential to become unresponsive due to the memory limits involved. Load balancing and trend prediction diminish the risks.
The portal
The majority of error messages seen by the end user via the error form start with:
org.sakaiproject.portal.api.PortalHandlerException: org.sakaiproject.tool.api.ToolException
The Sakai application captures exceptions that bubble up and wraps them in a top-level exception called PortalHandlerException. The exception that it catches is a Tool exception, a tool being Chat, Blog, and so on. The tool has caught an exception within its own responsibility, hence the ToolException. This is stating that there is a problem with a specific tool. Underneath these two lines of stack trace lie the details of the exact problem.
A full example is:
org.sakaiproject.portal.api.PortalHandlerException: org.sakaiproject.tool.api.ToolException at org.sakaiproject.portal.charon.SkinnableCharonPortal. doGet(SkinnableCharonPortal.java:767) caused by: org.sakaiproject.tool.api.ToolException at org.sakaiproject.site.tool.SiteAction. toolModeDispatch(SiteAction.java:10990) caused by: java.lang.reflect.InvocationTargetException at sun.reflect.DelegatingMethodAccessorImpl. invoke(DelegatingMethodAccessorImpl.java:25) caused by: org.springframework.dao.InvalidDataAccessResourceUsageException: could not execute query; nested exception is org.hibernate. exception.SQLGrammarException: could not execute query at org.springframework.orm.hibernate3.HibernateAccessor. convertHibernateAccessException(HibernateAccessor.java:412) caused by: org.hibernate.exception.SQLGrammarException: could not execute query at org.hibernate.exception.JDBCExceptionHelper.convert (JDBCExceptionHelper.java:43) caused by: java.sql.SQLException: ORA-00942: table or view does not exist
Reading the stack trace from the top tells you where in Sakai the issue is manifesting itself, in this case the Portal, and from the bottom upwards explains the exact cause. An error is returned by Oracle databases. JDBC is a low-level Java framework for database connectivity, and is being used by the Hibernate framework.
The database
Three database types are tested for common configuration settings during each QA cycle. The three types are the open source product MySQL; the mature commercial offering and industrial standard for many enterprises Oracle; and the in-memory database HSQL (http://hsqldb.org).
For performance reasons, the release notes recommend that you do not use HSQL for anything apart from demonstrations.
The database is core to Sakai. Any issues here are immediately felt by the end user. Much time and effort has gone into optimization. Absolutely not recommended, if you want to destroy your demonstration instance and generate gigabytes of log files within seconds then modify the database connection from the setting url@javax.sql.BaseDataSource=jdbc:hsqldb:file:${sakai.home}db/sakai.db, to the setting url@javax.sql.BaseDataSource=jdbc:xhsqldb:file:${sakai.home}db/sakai.db. What is happening is that the Spring and Hibernate frameworks, on which Sakai relies heavily, themselves call database connections via the Java JDBC framework (http://java.sun.com/docs/tutorials/tutorial/jdbc/overview/architecture.html). JDBC allows plugging in of drivers for different databases and you can switch between drivers using the URL setting. That is exactly what has just been done: changing from the hsqldb driver to a nonexistent xhsqldb driver. If you forget to place the database-specific driver's .jar file under the common/lib directory, you will find the same issue for the MySQL or Oracle drivers.
Any errors mentioned in the logs that have the word JDBC should not be taken lightly, as they pertain to the health of the database.
Both Spring and Hibernate abstract the underlying database details, so that developers have a uniform way of programming over a wide range of database types. However, sometimes subtle differences have the potential to cause problems. For example (http://confluence.sakaiproject.org/confluence/x/26c), by default, table names in MySQL are case insensitive on Windows and case sensitive on Unix systems. Portions of Sakai that attempt to access tables directly may specify table names in all uppercase or lowercase. The solution to this problem is to configure MySQL to think of table names as being case insensitive. This can be accomplished by editing /etc/my.cnf and adding the following:
lower_case_table_names=1
Search
Sometimes there are logged warnings that are purely cosmetic. However, stack traces give the impression that the system is shouting loudly for help. At the time of writing for Sakai 2.5.x and specifically for the in-memory database, the following error message is always generated at start up:
WARN BasicSqlService:1934 - Sql.ddl: missing resource: hsqldb/sakai_search_parallel.sql 14:27:40,601 ERROR TransactionSequenceImpl:106 - Failed to check transaction table java.sql.SQLException: Table not found in statement [select txid from search_transaction] at org.hsqldb.jdbc.Util.sqlException(Unknown Source) at org.hsqldb.jdbc.jdbcStatement.fetchResult(Unknown Source) @org.sakaiproject.content.api.ContentHostingService = vol1
What the error is stating is that a database table is missing. Next the error message java.sql.SQLException: Table not found in statement is generated by the application because the database table does not exist. Therefore, for the HSQL demonstration database transactions are broken for searching.
To turn search functionality off, set the following in
sakai/sakai.properties:search.enable=false
After restarting Sakai and trying search, you will see the following message returned by the application: The search tool is not enabled, please ask your administrator to
set search.enable = true in sakai.properties
sakai.properties
There are hundreds of properties that you can add to the global configuration file sakai/sakai.properties. The property names remain mostly stable between versions of Sakai. Occasionally, an unexpected change in a property name or function causes issues. The change is flagged early either through the testing cycle on the QA server network or later by the early adopters. In reaction, the community officer updates the release notes or developers modify the property name to fit its older purpose. Keeping track of the property list is a chore of great importance. However, new properties do emerge and supersede others. With over 500 configuration properties mentioned at http://bugs.sakaiproject.org/confluence/display/DOC/Sakai+Properties+Reference there is much detail in which you can make mistakes.
Never underestimate the capacity to make typing errors. (Shush do not let the editor hear that.) It is sometimes difficult to see the difference between 0 and o, and extra spaces hidden in a bunch of text are easy to miss at the best of times.
An historic example of the vulnerability of sakai.properties to potential typos is an issue that thankfully no longer exists (http://bugs.sakaiproject.org/jira/browse/SAK-9752).
Sakai can store resources such as videos, pictures, and documents either within the database itself or within the file system. If the system administrators have defined these two properties correctly: bodyPath@org.sakaiproject.content.api.ContentHostingService and bodyVolumes@org.sakaiproject.content.api.ContentHostingService then the binary files are stored on disk, otherwise they are stored in the database. The bodyPath defines the top-level directory where content is stored and the bodyVolumes is a comma-delimitated list of subdirectories. For example:
bodyPath@org.sakaiproject.content.api.ContentHostingService = ${sakai.home}db/bodyContent/
bodyVolumes@org.sakaiproject.content.api.ContentHostingService=vol1,vol2,vol3
The settings define three directories for storing content:
sakai/db/bodyContent/vol1 sakai/db/bodyContent/vol2 sakai/db/bodyContent/vol3
The bodyVolumes property allows content to be shared across disks or even across file servers mounted by NFS. However, historically, if spaces were accidently included in the list such as:
bodyVolumes@org.sakaiproject.content.api.ContentHostingService= vol1, vol2,vol3
then Sakai expects corresponding directories with spaces included.
Another issue for property configuration is that you need to think in terms of groups of properties rather than individual properties. One group of properties configures the database settings, another, the look and feel of Sakai, and yet another authentication. Looking at bodyPath and bodyVolumes, if you configure bodyPath alone then there are no subdirectories for binary content. All the content resides under the top-level directory. However, later, due to increased demand as users upload more and more files, you cannot add extra subdirectories. In other words, from the very beginning both properties need to be defined, or later on, as your system becomes more and more popular, you may have a storage scalability issue.
File permissions
It should not happen, but it does. Permissions on file systems can accidently be changed by errant processes such as tired system administrators working too hard, too fast on too many issues at once, or a poorly written shell script that has worked well for years, but breaks due to new circumstances in production.
If Sakai cannot write to specific parts of the file system, then content cannot be saved (if stored on the file system and not in the database). To make the situation more complex, metadata such as file name and location are stored in the database, so even if you fail to upload a file, then when the particular file is asked for again, the user will see a confusing file not found error java.io.FileNotFoundException. Therefore, if in the logs you see consistent java.io errors, then start checking file system permissions.
Class not found
If in the logs you see the error java.lang.ClassNotFoundException then that normally implies that a Java library is missing. For example if you configure your demonstration version of Sakai to use a MySQL database rather than the default in-memory database then you will see this error message:
java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
Each tool has its own space for loading in libraries. The file location is under the webapps/Toolname/WEB-INF/lib directory for JAR files and webapps/Toolname/WEB-INF/classes directory for compiled class files. These should be the first locations a system administrator checks for missing files. The directory common/lib is the location for Java libraries that need to be seen by all parts of the Sakai code base and where the database drivers need to be placed.
You can also generate a classNotFoundException when there are two copies of the same JAR file, but they are in different class loader locations in the same hierarchy within the same application. For example if you place your MySQL driver in common/lib and in a tools WEB_INF/lib directory.
Information sources
A handy source of information is the distribution lists, which are explained in detail in Tutorial 17. The lists contain a lot of questions and answers, many of which are relevant for issues associated with live deployments. There is a highly useable searchable index at http://www.nabble.com. On the front page of Nabble, there is a search form; if you type in the word Sakai, the returned results mention the various distribution lists followed by the most recent postings. Searching in this way, you will find more than 30,000 matching posts. The development list is the highest volume list; it has embedded within its contents a lot of hard-earned experience. Clicking on the link Sakai should take you to a page similar to the one shown in the figure below:
By selecting a forum and then searching, you can see if anyone has asked the community directly for help for any given problem.
For system administrators, the bug database (http://bugs.sakaiproject.org/jira) mentions more than 9,000 fixed issues or feature requests, with comments and a history of what was done to resolve the specific bug. The bug database is a central point for collecting and maintaining descriptions and status. Any individual is welcome to create an account and then report or comment on bugs.
|
The web site has an account creation link just underneath the login form. |
Each issue is linked to a given part of Sakai, such as the Chat or Blog tools, and the Sakai version number. If the issue, known as a "Jira" (after the name of the software the bug database runs on) has merit, then the Foundation's project coordinator assigns the task of cleaning up the code to a specific developer.
Not only are bugs reported and tracked within the bug database, but so too are feature requests. The community can vote on feature requests. If there is enough interest, then volunteers are assigned.
If you have an error message that you want to gather more information about, searching JIRA using the the quick search input on the top righthand side of the web site (next figure) will list the links to possibly related issues. Clicking on the most relevant link in the key column of the results table will direct you to the bug details page.
Searching for the term mysql mentions around 1000 fixed issues and for Oracle, 1200. This clearly shows that the QA process is active and developers are working hard at improving the code base.
Additional References
For instructions on Building Home page using Sakai, click here
Source
The source of this content is Chapter 13: Debugging of Sakai error messages of Sakai Courseware Management: The Official Guide by Alan Mark Berg , Michael Korcuska (Packt Publishing, 2009).
