1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
|
Last updated: 2017-04-08
The FreeBSD build will build most of the tools in Ceph.
Note that the (kernel) RBD dependant items will not work
I started looking into Ceph, because the HAST solution with CARP and
ggate did not really do what I was looking for. But I'm aiming for
running a Ceph storage cluster on storage nodes that are running ZFS.
In the end the cluster would be running bhyve on RBD disk that are stored in
Ceph.
The FreeBSD build will build most of the tools in Ceph.
Progress from last report:
==========================
Most important change:
- A port is submitted: net/ceph-devel.
Other improvements:
* A new ceph-devel update will be submitted in April
- Ceph-Fuse works, allowing to mount a CephFS on a FreeBSD system and do
some work on it.
- Ceph-disk prepare and activate work for FileStore on ZFS, allowing
easy creation of OSDs.
- RBD is actually buildable and can be used to manage RADOS BLOCK
DEVICEs.
- Most of the awkward dependencies on Linux-isms are deleted only
/bin/bash is there to stay.
Getting the FreeBSD work on Ceph:
=================================
pkg install net/ceph-devel
Or:
cd "place to work on this"
git clone https://github.com/wjwithagen/ceph.git
cd ceph
git checkout wip.FreeBSD
Building Ceph
=============
- Go and start building
./do_freebsd.sh
Parts not (yet) included:
=========================
- KRBD
Kernel Rados Block Devices is implemented in the Linux kernel
And perhaps ggated could be used as a template since it does some of
the same, other than just between 2 disks. And it has a userspace
counterpart.
- BlueStore.
FreeBSD and Linux have different AIO API, and that needs to be made
compatible Next to that is there discussion in FreeBSD about
aio_cancel not working for all devicetypes
- CephFS as native filesystem
(Ceph-fuse does work.)
Cython tries to access an internal field in dirent which does not
compile
Build Prerequisites
===================
Compiling and building Ceph is tested on 12-CURRENT, but I guess/expect
11-RELEASE will also work. And Clang is at 3.8.0.
It uses the CLANG toolset that is available, 3.7 is no longer tested,
but was working when that was with 11-CURRENT.
Clang 3.4 (on 10.2-STABLE) does not have all required capabilites to
compile everything
The following setup will get things running for FreeBSD:
This all require root privilidges.
- Install bash and link it in /bin
sudo pkg install bash
sudo ln -s /usr/local/bin/bash /bin/bash
Getting the FreeBSD work on Ceph:
=================================
- cd "place to work on this"
git clone https://github.com/wjwithagen/ceph.git
cd ceph
git checkout wip.FreeBSD.201702
Building Ceph
=============
- Go and start building
./do_freebsd.sh
Parts not (yet) included:
=========================
- KRBD
Kernel Rados Block Devices is implemented in the Linux kernel
It seems that there used to be a userspace implementation first.
And perhaps ggated could be used as a template since it does some of
the same, other than just between 2 disks. And it has a userspace
counterpart.
- BlueStore.
FreeBSD and Linux have different AIO API, and that needs to be made
compatible Next to that is there discussion in FreeBSD about
aio_cancel not working for all devicetypes
- CephFS
Cython tries to access an internal field in dirent which does not
compile
Tests that verify the correct working of the above are also excluded
from the testset
Tests not (yet) include:
=======================
- None, although some test can fail if running tests in parallel and there is
not enough swap. Then tests will start to fail in strange ways.
Task to do:
===========
- Build an automated test platform that will build ceph/master on
FreeBSD and report the results back to the Ceph developers. This will
increase the maintainability of the FreeBSD side of things.
Developers are signalled that they are using Linux-isms that will not
compile/run on FreeBSD Ceph has several projects for this: Jenkins,
teuthology, pulpito, ...
But even just a while { compile } loop and report the build data on a
static webpage would do for starters.
- Run integration tests to see if the FreeBSD daemons will work with a
Linux Ceph platform.
- Compile and test the user space RBD (Rados Block Device).
- Investigate and see if an in-kernel RBD device could be developed a la
'ggate'
- Investigate the keystore, which could be kernel embedded on Linux an
currently prevents building Cephfs and some other parts.
- Scheduler information is not used atm, because the schedulers work
rather different. But at a certain point in time, this would need some
attention:
in: ./src/common/Thread.cc
- Improve the FreeBSD /etc/rc.d initscripts in the Ceph stack. Both
for testing, but mainly for running Ceph on production machines.
Work on ceph-disk and ceph-deploy to make it more FreeBSD and ZFS
compatible.
- Build test-cluster and start running some of the teuthology integration
tests on these.
Teuthology want to build its own libvirt and that does not quite work
with all the packages FreeBSD already has in place. Lots of minute
details to figure out
- Design a vitual disk implementation that can be used with behyve and
attached to an RBD image.
|
