RPM .spec for Apache Kafka

kafka.spec

%define _noarch_libdir /usr/lib 

%define rel_ver 0.7.0

Summary: A high-throughput distributed messaging system.
Name: kafka
Version: %{rel_ver}
Release: 1
License: Apache License v2.0
Group: Applications/Databases
URL: http://incubator.apache.org/kafka/
Source0: http://people.apache.org/~nehanarkhede/kafka-0.7.0-incubating/kafka-0.7.0-incubating-src.tar.gz
BuildRoot: %{_tmppath}/%{name}-%{rel_ver}-%{release}-root
BuildRequires: java-1.6.0-openjdk-devel
Requires: java-1.6.0-openjdk
Requires(post): chkconfig initscripts
Requires(pre): chkconfig initscripts
AutoReqProv: no

%description

Apache Kafka is a distributed publish-subscribe messaging system. It
is designed to support the following:

* Persistent messaging with O(1) disk structures that provide constant
  time performance even with many TB of stored messages.

* High-throughput: even with very modest hardware Kafka can support
  hundreds of thousands of messages per second.

* Explicit support for partitioning messages over Kafka servers and
  distributing consumption over a cluster of consumer machines while
  maintaining per-partition ordering semantics.

* Support for parallel data load into Hadoop.

Kafka provides a publish-subscribe solution that can handle all
activity stream data and processing on a consumer-scale web site. This
kind of activity (page views, searches, and other user actions) are a
key ingredient in many of the social feature on the modern web. This
data is typically handled by "logging" and ad hoc log aggregation
solutions due to the throughput requirements. This kind of ad hoc
solution is a viable solution to providing logging data to an offline
analysis system like Hadoop, but is very limiting for building
real-time processing. Kafka aims to unify offline and online
processing by providing a mechanism for parallel load into Hadoop as
well as the ability to partition real-time consumption over a cluster
of machines.

The use for activity stream processing makes Kafka comparable to
Facebook's Scribe or Apache Flume (incubating), though the
architecture and primitives are very different for these systems and
make Kafka more comparable to a traditional messaging system. See our
design page for more details.

%define _kafka_noarch_libdir %{_noarch_libdir}/kafka

%prep
%setup -q -n kafka-%{rel_ver}-incubating-src

%{__cat} <<EOF >sysconfig
KAFKA_OPTS="-Xmx512M -server -Dlog4j.configuration=file:/etc/kafka/log4j.properties"
EOF

%{__cat} <<EOF >log4j.properties
#
# Kafka Logging Configuration
#

# Format is "<default threshold> (, <appender>)+

# DEFAULT: console appender only
#log4j.rootLogger=INFO, CONSOLE

log4j.rootLogger=INFO, ROLLINGFILE

# Example with rolling log file
#log4j.rootLogger=DEBUG, CONSOLE, ROLLINGFILE

# Example with rolling log file and tracing
#log4j.rootLogger=TRACE, CONSOLE, ROLLINGFILE, TRACEFILE

# Turn on all our debugging info
#log4j.logger.kafka=INFO
#log4j.logger.org.I0Itec.zkclient.ZkClient=DEBUG

#
# Log INFO level and above messages to the console
#
log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender
log4j.appender.CONSOLE.Threshold=INFO
log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout
log4j.appender.CONSOLE.layout.ConversionPattern=%d{ISO8601} - %-5p [%t:%C{1}@%L] - %m%n

#
# Add ROLLINGFILE to rootLogger to get log file output
#    Log DEBUG level and above messages to a log file
log4j.appender.ROLLINGFILE=org.apache.log4j.RollingFileAppender
log4j.appender.ROLLINGFILE.Threshold=DEBUG
log4j.appender.ROLLINGFILE.File=/var/log/kafka/kafka.log

# Max log file size of 10MB
log4j.appender.ROLLINGFILE.MaxFileSize=10MB
# uncomment the next line to limit number of backup files
#log4j.appender.ROLLINGFILE.MaxBackupIndex=10

log4j.appender.ROLLINGFILE.layout=org.apache.log4j.PatternLayout
log4j.appender.ROLLINGFILE.layout.ConversionPattern=%d{ISO8601} - %-5p [%t:%C{1}@%L] - %m%n

#
# Add TRACEFILE to rootLogger to get log file output
#    Log DEBUG level and above messages to a log file
log4j.appender.TRACEFILE=org.apache.log4j.FileAppender
log4j.appender.TRACEFILE.Threshold=TRACE
log4j.appender.TRACEFILE.File=/var/log/kafka/kafka_trace.log

log4j.appender.TRACEFILE.layout=org.apache.log4j.PatternLayout
### Notice we are including log4j's NDC here (%x)
log4j.appender.TRACEFILE.layout.ConversionPattern=%d{ISO8601} - %-5p [%t:%C{1}@%L][%x] - %m%n
EOF

%{__cat} <<EOF >server.properties
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
# 
#    http://www.apache.org/licenses/LICENSE-2.0
# 
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# see kafka.server.KafkaConfig for additional details and defaults

############################# Server Basics #############################

# The id of the broker. This must be set to a unique integer for each
# broker.
brokerid=0

# Hostname the broker will advertise to consumers. If not set, kafka
# will use the value returned from InetAddress.getLocalHost().  If
# there are multiple interfaces getLocalHost may not be what you want.
#hostname=

############################# Socket Server Settings #############################

# The port the socket server listens on
port=9092

# The number of processor threads the socket server uses for receiving and answering requests. 
# Defaults to the number of cores on the machine
num.threads=8

# The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer=1048576

# The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer=1048576

# The maximum size of a request that the socket server will accept (protection against OOM)
max.socket.request.bytes=104857600

############################# Log Basics #############################

# The directory under which to store log files
log.dir=/var/lib/kafka

# The number of logical partitions per topic per server. More partitions allow greater parallelism
# for consumption, but also mean more files.
num.partitions=1

# Overrides for for the default given by num.partitions on a per-topic basis
#topic.partition.count.map=topic1:3, topic2:4

############################# Log Flush Policy #############################

# The following configurations control the flush of data to disk. This is the most
# important performance knob in kafka.
# There are a few important trade-offs here:
#    1. Durability: Unflushed data is at greater risk of loss in the event of a crash.
#    2. Latency: Data is not made available to consumers until it is flushed (which adds latency).
#    3. Throughput: The flush is generally the most expensive operation. 
# The settings below allow one to configure the flush policy to flush data after a period of time or
# every N messages (or both). This can be done globally and overridden on a per-topic basis.

# The number of messages to accept before forcing a flush of data to disk
log.flush.interval=10000

# The maximum amount of time a message can sit in a log before we force a flush
log.default.flush.interval.ms=1000

# Per-topic overrides for log.default.flush.interval.ms
#topic.flush.intervals.ms=topic1:1000, topic2:3000

# The interval (in ms) at which logs are checked to see if they need to be flushed to disk.
log.default.flush.scheduler.interval.ms=1000

############################# Log Retention Policy #############################

# The following configurations control the disposal of log segments. The policy can
# be set to delete segments after a period of time, or after a given size has accumulated.
# A segment will be deleted whenever *either* of these criteria are met. Deletion always happens
# from the end of the log.

# The minimum age of a log file to be eligible for deletion
log.retention.hours=168

# A size-based retention policy for logs. Segments are pruned from the log as long as the remaining
# segments don't drop below log.retention.size.
#log.retention.size=1073741824

# The maximum size of a log segment file. When this size is reached a new log segment will be created.
log.file.size=536870912

# The interval at which log segments are checked to see if they can be deleted according 
# to the retention policies
log.cleanup.interval.mins=1

############################# Zookeeper #############################

# Enable connecting to zookeeper
enable.zookeeper=true

# Zk connection string (see zk docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zk.connect=localhost:2181

# Timeout in ms for connecting to zookeeper
zk.connectiontimeout.ms=1000000
EOF

%{__cat} <<'EOF' >sysv
#!/bin/bash
#
# Init file for Apache Kafka
#
# chkconfig: 35 85 15
# description: Apache Kafka is a distributed publish-subscribe messaging system
# processname: kafka
# config: /etc/kafka/servers.properties
# pidfile: /var/run/kafa.pid

. /etc/rc.d/init.d/functions

SYSCONFIG="/etc/sysconfig/kafka"
KAFKA_BASEDIR=${KAFKA_BASEDIR-/usr/lib/kafka}
PIDFILE="/var/run/kafka.pid"

. "$SYSCONFIG" || true

CMD="$KAFKA_BASEDIR/bin/kafka-run-class.sh kafka.Kafka /etc/kafka/server.properties & echo \$! > $PIDFILE"

start()
{
  echo -n $"Starting kafka: "
  touch $PIDFILE && chown kafka $PIDFILE
  export KAFKA_OPTS
  daemon --user kafka --pidfile $PIDFILE "$CMD"
  RETVAL=$?
  [ $RETVAL -eq 0 ] && touch /var/lock/subsys/kafka && success || failure
  echo
  return $RETVAL
}

stop()
{
  echo -n $"Stopping kafka: "
  killproc -p $PIDFILE kafka
  RETVAL=$?
  echo
  [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/kafka
  return $RETVAL
}

restart () {
  stop
  start
}

RETVAL=0

case "$1" in
  start)
    start
    ;;
  stop)
    stop
    ;;
  restart|reload|force-reload)
    restart
    ;;
  condrestart)
    [ -f /var/lock/subsys/kafka ] && restart || :
    ;;
  status)
    status -p $PIDFILE kafka
    RETVAL=$?
    ;;
  *)
    echo "Usage: $0 {start|stop|status|restart|reload|force-reload|condrestart}"
    RETVAL=1
esac

exit $RETVAL
EOF

%build
./sbt update
./sbt package

head -n -1 bin/kafka-run-class.sh > run-class.sh
echo "exec \$JAVA \$KAFKA_OPTS \$KAFKA_JMX_OPTS -cp \$CLASSPATH \$@" >> run-class.sh
mv run-class.sh bin/kafka-run-class.sh
chmod +x bin/kafka-run-class.sh

%install
rm -rf %{buildroot}
install -p -d %{buildroot}%{_kafka_noarch_libdir}
cp -r bin core lib lib_managed project %{buildroot}%{_kafka_noarch_libdir}

mkdir -p %{buildroot}%{_sysconfdir}/kafka
install -p -D -m 644 server.properties %{buildroot}%{_sysconfdir}/kafka
install -p -D -m 644 log4j.properties %{buildroot}%{_sysconfdir}/kafka
install -p -D -m 644 sysconfig %{buildroot}%{_sysconfdir}/sysconfig/kafka
install -p -D -m 755 sysv %{buildroot}%{_initrddir}/kafka

install -d %{buildroot}%{_localstatedir}/lib/kafka
install -d %{buildroot}%{_localstatedir}/log/kafka

%clean
rm -rf %{buildroot}

%files
%defattr(-,root,root,-)
%doc README.md LICENSE NOTICE DISCLAIMER
%dir %attr(0750, kafka, kafka) %{_localstatedir}/lib/kafka
%dir %attr(0750, kafka, kafka) %{_localstatedir}/log/kafka
%{_kafka_noarch_libdir}
%{_initrddir}/kafka
%config(noreplace) %{_sysconfdir}/sysconfig/kafka
%config(noreplace) %{_sysconfdir}/kafka

%pre
getent group kafka >/dev/null || groupadd -r kafka
getent passwd kafka >/dev/null || useradd -r -g kafka -d / -s /sbin/nologin kafka
exit 0

%post
/sbin/chkconfig --add kafka

%preun
if [ $1 = 0 ] ; then
  /sbin/service kafka stop >/dev/null 2>&1
  /sbin/chkconfig --del kafka
fi

%postun
if [ "$1" -ge "1" ] ; then
  /sbin/service kafka condrestart >/dev/null 2>&1 || :
fi

%changelog
* Fri Mar 16 2012  Johan Rydberg <[email protected]> - 0.7.0-1
- Update to 0.7.0