[dm-devel] [PATCH for-3.14] Add dm-writeboost (log-structured caching target)
Akira Hayakawa
ruby.wktk at gmail.com
Sat Jan 11 07:36:02 UTC 2014
dm-writeboost is an another cache target like dm-cache and bcache.
The biggest difference from existing cache softwares is that
it focuses on bursty writes.
dm-writeboost first writes the data to RAM buffer and makes a
log containing both data and their metadata.
The log is written to the cache device in log-structured manner.
The fact that the log contains metadata of the data blocks makes
dm-writeboost is robust for power fault. It can replay the log
after crash.
Signed-off-by: Akira Hayakawa <ruby.wktk at gmail.com>
---
Documentation/device-mapper/dm-writeboost.txt | 161 +++
drivers/md/Kconfig | 8 +
drivers/md/Makefile | 3 +
drivers/md/dm-writeboost-daemon.c | 520 ++++++++++
drivers/md/dm-writeboost-daemon.h | 40 +
drivers/md/dm-writeboost-metadata.c | 1352 +++++++++++++++++++++++++
drivers/md/dm-writeboost-metadata.h | 51 +
drivers/md/dm-writeboost-target.c | 1258 +++++++++++++++++++++++
drivers/md/dm-writeboost.h | 464 +++++++++
9 files changed, 3857 insertions(+)
create mode 100644 Documentation/device-mapper/dm-writeboost.txt
create mode 100644 drivers/md/dm-writeboost-daemon.c
create mode 100644 drivers/md/dm-writeboost-daemon.h
create mode 100644 drivers/md/dm-writeboost-metadata.c
create mode 100644 drivers/md/dm-writeboost-metadata.h
create mode 100644 drivers/md/dm-writeboost-target.c
create mode 100644 drivers/md/dm-writeboost.h
diff --git a/Documentation/device-mapper/dm-writeboost.txt b/Documentation/device-mapper/dm-writeboost.txt
new file mode 100644
index 0000000..0161663
--- /dev/null
+++ b/Documentation/device-mapper/dm-writeboost.txt
@@ -0,0 +1,161 @@
+dm-writeboost
+=============
+Writeboost target provides log-structured caching.
+It batches random writes into a big sequential write to a cache device.
+
+It is like dm-cache as a cache target but the difference is that Writeboost
+focuses on bursty writes and the lifetime of the SSD cache device.
+
+More documents and tests are available in
+https://github.com/akiradeveloper/dm-writeboost
+
+Design
+======
+There are 1 foreground and 6 background processes.
+
+Foreground
+----------
+It accepts bios and stores the write data to RAM buffer.
+When the buffer is full, it creates a "flush job" and queues it.
+
+Background
+----------
+* wbflusher (Writeboost flusher)
+Executes a flush job.
+wbflusher exploits workqueue mechanism and may run in parallel.
+It exhibits the sysfs (/sys/bus/workqueue/devices/wbflusher)
+to control the behavior.
+
+* Barrier deadline worker
+Barrier flags such as REQ_FUA and REQ_FLUSH are acked lazily.
+Immediately handling these bios badly deteriorate the throughput.
+Bios with these flags are queued and forcefully processed at worst
+within `barrier_deadline_ms` period.
+
+* Migrate Daemon
+It migrates, or writes back, cache data to backing store.
+
+If `allow_migrate` is true, it migrates without impending situation.
+Being in impending situation is that there are no room in cache device
+for writing more flush jobs.
+
+Migration is done batching `nr_max_batched_migration` segments at maximum
+at a time. Thus, unlike existing I/O scheduler, two dirty writes close in
+positional space but distant in time space can be merged. Writetboost is
+also a extension of I/O scheduler.
+
+* Migration Modulator
+Migration while the backing store is heavily loaded grows the device queue
+longer and affects the read from the backing store.
+Migration modulator surveils the load of the backing store and turns on/off
+the migration by switching `allow_migrate`.
+
+* Superblock Recorder
+Superblock is a last sector of first 1MB region in cache device containing
+what id of the segment lastly migrated. This daemon periodically updates
+the region every `update_record_interval` seconds.
+
+* Sync Daemon
+This daemon forcefully writes out all the dirty data persistently every
+`sync_interval` seconds. Some careful users want to make all the writes
+persistent periodically.
+
+Target Interface
+================
+All the operations are via dmsetup command.
+
+Constructor
+-----------
+<type>
+<essential args>*
+<#optional args> <optional args>*
+<#tunable args> <tunable args>* (see 'Message')
+
+Optionals are tunables are unordered lists of Key-Value pairs.
+
+Essential args and optional args are different for different buffer type.
+
+<type> (The type of the RAM buffer)
+0: volatile RAM buffer (DRAM)
+1: non-volatile buffer with a block I/F
+2: non-volatile buffer with PRAM I/F
+
+Currently, only type 0 is supported.
+
+Type 0
+------
+<essential args>
+backing_dev : Slow device holding original data blocks.
+cache_dev : Fast device holding cached data and its metadata.
+
+<optional args>
+segment_size_order : The size of RAM buffer
+ 1 << n (sectors), 4 <= n <= 10
+ default 7
+rambuf_pool_amount : The amount of the RAM buffer pool (kB).
+ Too fewer amount may cause waiting for new buffer
+ to become available again. But too much doesn't
+ benefit the performance.
+ default 2048
+
+Note that cache device is re-formatted if the first sector of the cache
+device is zeroed out.
+
+Status
+------
+<cursor pos>
+<#cache blocks>
+<#segments>
+<current id>
+<lastly flushed id>
+<lastly migrated id>
+<#dirty cache blocks>
+<stat (w/r) x (hit/miss) x (on buffer?) x (fullsize?)>
+<#not full flushed>
+<#tunable args> [tunable args]
+
+Messages
+--------
+You can tune up the behavior of writeboost via message interface.
+
+* barrier_deadline_ms (ms)
+Default: 3
+All the bios with barrier flags like REQ_FUA or REQ_FLUSH
+are guaranteed to be acked within this deadline.
+
+* allow_migrate (bool)
+Default: 1
+Set to 1 to start migration.
+
+* enable_migration_modulator (bool) and
+ migrate_threshold (%)
+Default: 1 and 70
+Set to 1 to run migration modulator.
+Migration modulator surveils the load of backing store and sets the
+migration started if the load is lower than the `migrate_threshold`.
+
+* nr_max_batched_migration (int)
+Default: 1MB / segment size
+Number of segments to migrate at a time.
+Set higher value to fully exploit the capacily of the backing store.
+Even a single HDD is capable of processing 1MB/sec random writes so
+the default value is set to 1MB / segment size. Set higher value if
+you use RAID-ed drive as the backing store.
+
+* update_record_interval (sec)
+Default: 60
+The superblock record is updated every update_record_interval seconds.
+
+* sync_interval (sec)
+Default: 60
+All the dirty writes are guaranteed to be persistent every this interval.
+
+Example
+=======
+dmsetup create writeboost-vol --table "0 ${sz} 0 writeboost ${BACKING} {CACHE}"
+dmsetup create writeboost-vol --table "0 ${sz} 0 writeboost ${BACKING} {CACHE} \
+ 4 rambuf_pool_amount 8192 segment_size_order 8 \
+ 2 allow_migrate 1"
+dmsetup create writeboost-vol --table "0 ${sz} 0 writeboost ${BACKING} {CACHE} \
+ 0 \
+ 2 allow_migrate 1"
diff --git a/drivers/md/Kconfig b/drivers/md/Kconfig
index f2ccbc3..65a6d95 100644
--- a/drivers/md/Kconfig
+++ b/drivers/md/Kconfig
@@ -290,6 +290,14 @@ config DM_CACHE_CLEANER
A simple cache policy that writes back all data to the
origin. Used when decommissioning a dm-cache.
+config DM_WRITEBOOST
+ tristate "Log-structured Caching (EXPERIMENTAL)"
+ depends on BLK_DEV_DM
+ default y
+ ---help---
+ A cache layer that batches random writes into a big sequential
+ write to a cache device in log-structured manner.
+
config DM_MIRROR
tristate "Mirror target"
depends on BLK_DEV_DM
diff --git a/drivers/md/Makefile b/drivers/md/Makefile
index 2acc43f..6db61ce 100644
--- a/drivers/md/Makefile
+++ b/drivers/md/Makefile
@@ -14,6 +14,8 @@ dm-thin-pool-y += dm-thin.o dm-thin-metadata.o
dm-cache-y += dm-cache-target.o dm-cache-metadata.o dm-cache-policy.o
dm-cache-mq-y += dm-cache-policy-mq.o
dm-cache-cleaner-y += dm-cache-policy-cleaner.o
+dm-writeboost-y += dm-writeboost-target.o dm-writeboost-metadata.o \
+ dm-writeboost-daemon.o
md-mod-y += md.o bitmap.o
raid456-y += raid5.o
@@ -52,6 +54,7 @@ obj-$(CONFIG_DM_VERITY) += dm-verity.o
obj-$(CONFIG_DM_CACHE) += dm-cache.o
obj-$(CONFIG_DM_CACHE_MQ) += dm-cache-mq.o
obj-$(CONFIG_DM_CACHE_CLEANER) += dm-cache-cleaner.o
+obj-$(CONFIG_DM_WRITEBOOST) += dm-writeboost.o
ifeq ($(CONFIG_DM_UEVENT),y)
dm-mod-objs += dm-uevent.o
diff --git a/drivers/md/dm-writeboost-daemon.c b/drivers/md/dm-writeboost-daemon.c
new file mode 100644
index 0000000..5ea1300
--- /dev/null
+++ b/drivers/md/dm-writeboost-daemon.c
@@ -0,0 +1,520 @@
+/*
+ * Copyright (C) 2012-2014 Akira Hayakawa <ruby.wktk at gmail.com>
+ *
+ * This file is released under the GPL.
+ */
+
+#include "dm-writeboost.h"
+#include "dm-writeboost-metadata.h"
+#include "dm-writeboost-daemon.h"
+
+/*----------------------------------------------------------------*/
+
+static void update_barrier_deadline(struct wb_device *wb)
+{
+ mod_timer(&wb->barrier_deadline_timer,
+ jiffies + msecs_to_jiffies(ACCESS_ONCE(wb->barrier_deadline_ms)));
+}
+
+void queue_barrier_io(struct wb_device *wb, struct bio *bio)
+{
+ mutex_lock(&wb->io_lock);
+ bio_list_add(&wb->barrier_ios, bio);
+ mutex_unlock(&wb->io_lock);
+
+ if (!timer_pending(&wb->barrier_deadline_timer))
+ update_barrier_deadline(wb);
+}
+
+void barrier_deadline_proc(unsigned long data)
+{
+ struct wb_device *wb = (struct wb_device *) data;
+ schedule_work(&wb->barrier_deadline_work);
+}
+
+void flush_barrier_ios(struct work_struct *work)
+{
+ struct wb_device *wb = container_of(
+ work, struct wb_device, barrier_deadline_work);
+
+ if (bio_list_empty(&wb->barrier_ios))
+ return;
+
+ atomic64_inc(&wb->count_non_full_flushed);
+ flush_current_buffer(wb);
+}
+
+/*----------------------------------------------------------------*/
+
+static void
+process_deferred_barriers(struct wb_device *wb, struct flush_job *job)
+{
+ int r = 0;
+ bool has_barrier = !bio_list_empty(&job->barrier_ios);
+
+ /*
+ * Make all the data until now persistent.
+ */
+ if (has_barrier)
+ IO(blkdev_issue_flush(wb->cache_dev->bdev, GFP_NOIO, NULL));
+
+ /*
+ * Ack the chained barrier requests.
+ */
+ if (has_barrier) {
+ struct bio *bio;
+ while ((bio = bio_list_pop(&job->barrier_ios))) {
+ LIVE_DEAD(
+ bio_endio(bio, 0),
+ bio_endio(bio, -EIO)
+ );
+ }
+ }
+
+ if (has_barrier)
+ update_barrier_deadline(wb);
+}
+
+void flush_proc(struct work_struct *work)
+{
+ int r = 0;
+
+ struct flush_job *job = container_of(work, struct flush_job, work);
+
+ struct wb_device *wb = job->wb;
+ struct segment_header *seg = job->seg;
+
+ struct dm_io_request io_req = {
+ .client = wb_io_client,
+ .bi_rw = WRITE,
+ .notify.fn = NULL,
+ .mem.type = DM_IO_KMEM,
+ .mem.ptr.addr = job->rambuf->data,
+ };
+ struct dm_io_region region = {
+ .bdev = wb->cache_dev->bdev,
+ .sector = seg->start_sector,
+ .count = (seg->length + 1) << 3,
+ };
+
+ /*
+ * The actual write requests to the cache device are not serialized.
+ * They may perform in parallel.
+ */
+ IO(dm_safe_io(&io_req, 1, ®ion, NULL, false));
+
+ /*
+ * Deferred ACK for barrier requests
+ * To serialize barrier ACK in logging we wait for the previous
+ * segment to be persistently written (if needed).
+ */
+ wait_for_flushing(wb, SUB_ID(seg->id, 1));
+
+ process_deferred_barriers(wb, job);
+
+ /*
+ * We can count up the last_flushed_segment_id only after segment
+ * is written persistently. Counting up the id is serialized.
+ */
+ atomic64_inc(&wb->last_flushed_segment_id);
+ wake_up_interruptible(&wb->flush_wait_queue);
+
+ mempool_free(job, wb->flush_job_pool);
+}
+
+void wait_for_flushing(struct wb_device *wb, u64 id)
+{
+ wait_event_interruptible(wb->flush_wait_queue,
+ atomic64_read(&wb->last_flushed_segment_id) >= id);
+}
+
+/*----------------------------------------------------------------*/
+
+static void migrate_endio(unsigned long error, void *context)
+{
+ struct wb_device *wb = context;
+
+ if (error)
+ atomic_inc(&wb->migrate_fail_count);
+
+ if (atomic_dec_and_test(&wb->migrate_io_count))
+ wake_up_interruptible(&wb->migrate_io_wait_queue);
+}
+
+/*
+ * Asynchronously submit the segment data at position k in the migrate buffer.
+ * Batched migration first collects all the segments to migrate into a migrate buffer.
+ * So, there are a number of segment data in the migrate buffer.
+ * This function submits the one in position k.
+ */
+static void submit_migrate_io(struct wb_device *wb, struct segment_header *seg,
+ size_t k)
+{
+ int r = 0;
+
+ size_t a = wb->nr_caches_inseg * k;
+ void *p = wb->migrate_buffer + (wb->nr_caches_inseg << 12) * k;
+
+ u8 i;
+ for (i = 0; i < seg->length; i++) {
+ unsigned long offset = i << 12;
+ void *base = p + offset;
+
+ struct metablock *mb = seg->mb_array + i;
+ u8 dirty_bits = *(wb->dirtiness_snapshot + (a + i));
+ if (!dirty_bits)
+ continue;
+
+ if (dirty_bits == 255) {
+ void *addr = base;
+ struct dm_io_request io_req_w = {
+ .client = wb_io_client,
+ .bi_rw = WRITE,
+ .notify.fn = migrate_endio,
+ .notify.context = wb,
+ .mem.type = DM_IO_VMA,
+ .mem.ptr.vma = addr,
+ };
+ struct dm_io_region region_w = {
+ .bdev = wb->origin_dev->bdev,
+ .sector = mb->sector,
+ .count = 1 << 3,
+ };
+ IO(dm_safe_io(&io_req_w, 1, ®ion_w, NULL, false));
+ } else {
+ u8 j;
+ for (j = 0; j < 8; j++) {
+ struct dm_io_request io_req_w;
+ struct dm_io_region region_w;
+
+ void *addr = base + (j << SECTOR_SHIFT);
+ bool bit_on = dirty_bits & (1 << j);
+ if (!bit_on)
+ continue;
+
+ io_req_w = (struct dm_io_request) {
+ .client = wb_io_client,
+ .bi_rw = WRITE,
+ .notify.fn = migrate_endio,
+ .notify.context = wb,
+ .mem.type = DM_IO_VMA,
+ .mem.ptr.vma = addr,
+ };
+ region_w = (struct dm_io_region) {
+ .bdev = wb->origin_dev->bdev,
+ .sector = mb->sector + j,
+ .count = 1,
+ };
+ IO(dm_safe_io(&io_req_w, 1, ®ion_w, NULL, false));
+ }
+ }
+ }
+}
+
+static void memorize_data_to_migrate(struct wb_device *wb,
+ struct segment_header *seg, size_t k)
+{
+ int r = 0;
+
+ void *p = wb->migrate_buffer + (wb->nr_caches_inseg << 12) * k;
+ struct dm_io_request io_req_r = {
+ .client = wb_io_client,
+ .bi_rw = READ,
+ .notify.fn = NULL,
+ .mem.type = DM_IO_VMA,
+ .mem.ptr.vma = p,
+ };
+ struct dm_io_region region_r = {
+ .bdev = wb->cache_dev->bdev,
+ .sector = seg->start_sector + (1 << 3),
+ .count = seg->length << 3,
+ };
+ IO(dm_safe_io(&io_req_r, 1, ®ion_r, NULL, false));
+}
+
+/*
+ * We first memorize the snapshot of the dirtiness in the segments.
+ * The snapshot dirtiness is dirtier than that of any future moment
+ * because it is only monotonously decreasing after flushed.
+ * Therefore, we will migrate the possible dirtiest state of the
+ * segments which won't lose any dirty data.
+ */
+static void memorize_metadata_to_migrate(struct wb_device *wb, struct segment_header *seg,
+ size_t k, size_t *migrate_io_count)
+{
+ u8 i, j;
+
+ struct metablock *mb;
+ size_t a = wb->nr_caches_inseg * k;
+
+ /*
+ * We first memorize the dirtiness of the metablocks.
+ * Dirtiness may decrease while we run through the migration code
+ * and it may cause corruption.
+ */
+ for (i = 0; i < seg->length; i++) {
+ mb = seg->mb_array + i;
+ *(wb->dirtiness_snapshot + (a + i)) = read_mb_dirtiness(wb, seg, mb);
+ }
+
+ for (i = 0; i < seg->length; i++) {
+ u8 dirty_bits = *(wb->dirtiness_snapshot + (a + i));
+
+ if (!dirty_bits)
+ continue;
+
+ if (dirty_bits == 255) {
+ (*migrate_io_count)++;
+ } else {
+ for (j = 0; j < 8; j++) {
+ if (dirty_bits & (1 << j))
+ (*migrate_io_count)++;
+ }
+ }
+ }
+}
+
+/*
+ * Memorize the dirtiness snapshot and count up the number of io to migrate.
+ */
+static void memorize_dirty_state(struct wb_device *wb, struct segment_header *seg,
+ size_t k, size_t *migrate_io_count)
+{
+ memorize_data_to_migrate(wb, seg, k);
+ memorize_metadata_to_migrate(wb, seg, k, migrate_io_count);
+}
+
+static void cleanup_segment(struct wb_device *wb, struct segment_header *seg)
+{
+ u8 i;
+ for (i = 0; i < seg->length; i++) {
+ struct metablock *mb = seg->mb_array + i;
+ cleanup_mb_if_dirty(wb, seg, mb);
+ }
+}
+
+static void transport_emigrates(struct wb_device *wb)
+{
+ int r;
+ struct segment_header *seg;
+ size_t k, migrate_io_count = 0;
+
+ for (k = 0; k < wb->num_emigrates; k++) {
+ seg = *(wb->emigrates + k);
+ memorize_dirty_state(wb, seg, k, &migrate_io_count);
+ }
+
+migrate_write:
+ atomic_set(&wb->migrate_io_count, migrate_io_count);
+ atomic_set(&wb->migrate_fail_count, 0);
+
+ for (k = 0; k < wb->num_emigrates; k++) {
+ seg = *(wb->emigrates + k);
+ submit_migrate_io(wb, seg, k);
+ }
+
+ LIVE_DEAD(
+ wait_event_interruptible(wb->migrate_io_wait_queue,
+ !atomic_read(&wb->migrate_io_count)),
+ atomic_set(&wb->migrate_io_count, 0));
+
+ if (atomic_read(&wb->migrate_fail_count)) {
+ WBWARN("%u writebacks failed. retry",
+ atomic_read(&wb->migrate_fail_count));
+ goto migrate_write;
+ }
+ BUG_ON(atomic_read(&wb->migrate_io_count));
+
+ /*
+ * We clean up the metablocks because there is no reason
+ * to leave the them dirty.
+ */
+ for (k = 0; k < wb->num_emigrates; k++) {
+ seg = *(wb->emigrates + k);
+ cleanup_segment(wb, seg);
+ }
+
+ /*
+ * We must write back a segments if it was written persistently.
+ * Nevertheless, we betray the upper layer.
+ * Remembering which segment is persistent is too expensive
+ * and furthermore meaningless.
+ * So we consider all segments are persistent and write them back
+ * persistently.
+ */
+ IO(blkdev_issue_flush(wb->origin_dev->bdev, GFP_NOIO, NULL));
+}
+
+static void do_migrate_proc(struct wb_device *wb)
+{
+ u32 i, nr_mig_candidates, nr_mig, nr_max_batch;
+ struct segment_header *seg;
+
+ bool start_migrate = ACCESS_ONCE(wb->allow_migrate) ||
+ ACCESS_ONCE(wb->urge_migrate) ||
+ ACCESS_ONCE(wb->force_drop);
+
+ if (!start_migrate) {
+ schedule_timeout_interruptible(msecs_to_jiffies(1000));
+ return;
+ }
+
+ nr_mig_candidates = atomic64_read(&wb->last_flushed_segment_id) -
+ atomic64_read(&wb->last_migrated_segment_id);
+
+ if (!nr_mig_candidates) {
+ schedule_timeout_interruptible(msecs_to_jiffies(1000));
+ return;
+ }
+
+ nr_max_batch = ACCESS_ONCE(wb->nr_max_batched_migration);
+ if (wb->nr_cur_batched_migration != nr_max_batch)
+ try_alloc_migration_buffer(wb, nr_max_batch);
+ nr_mig = min(nr_mig_candidates, wb->nr_cur_batched_migration);
+
+ /*
+ * Store emigrates
+ */
+ for (i = 0; i < nr_mig; i++) {
+ seg = get_segment_header_by_id(wb,
+ atomic64_read(&wb->last_migrated_segment_id) + 1 + i);
+ *(wb->emigrates + i) = seg;
+ }
+ wb->num_emigrates = nr_mig;
+ transport_emigrates(wb);
+
+ atomic64_add(nr_mig, &wb->last_migrated_segment_id);
+ wake_up_interruptible(&wb->migrate_wait_queue);
+}
+
+int migrate_proc(void *data)
+{
+ struct wb_device *wb = data;
+ while (!kthread_should_stop())
+ do_migrate_proc(wb);
+ return 0;
+}
+
+/*
+ * Wait for a segment to be migrated.
+ * After migrated the metablocks in the segment are clean.
+ */
+void wait_for_migration(struct wb_device *wb, u64 id)
+{
+ wb->urge_migrate = true;
+ wake_up_process(wb->migrate_daemon);
+ wait_event_interruptible(wb->migrate_wait_queue,
+ atomic64_read(&wb->last_migrated_segment_id) >= id);
+ wb->urge_migrate = false;
+}
+
+/*----------------------------------------------------------------*/
+
+int modulator_proc(void *data)
+{
+ struct wb_device *wb = data;
+
+ struct hd_struct *hd = wb->origin_dev->bdev->bd_part;
+ unsigned long old = 0, new, util;
+ unsigned long intvl = 1000;
+
+ while (!kthread_should_stop()) {
+ new = jiffies_to_msecs(part_stat_read(hd, io_ticks));
+
+ if (!ACCESS_ONCE(wb->enable_migration_modulator))
+ goto modulator_update;
+
+ util = div_u64(100 * (new - old), 1000);
+
+ if (util < ACCESS_ONCE(wb->migrate_threshold))
+ wb->allow_migrate = true;
+ else
+ wb->allow_migrate = false;
+
+modulator_update:
+ old = new;
+
+ schedule_timeout_interruptible(msecs_to_jiffies(intvl));
+ }
+ return 0;
+}
+
+/*----------------------------------------------------------------*/
+
+static void update_superblock_record(struct wb_device *wb)
+{
+ int r = 0;
+
+ struct superblock_record_device o;
+ void *buf;
+ struct dm_io_request io_req;
+ struct dm_io_region region;
+
+ o.last_migrated_segment_id =
+ cpu_to_le64(atomic64_read(&wb->last_migrated_segment_id));
+
+ buf = mempool_alloc(wb->buf_1_pool, GFP_NOIO | __GFP_ZERO);
+ memcpy(buf, &o, sizeof(o));
+
+ io_req = (struct dm_io_request) {
+ .client = wb_io_client,
+ .bi_rw = WRITE_FUA,
+ .notify.fn = NULL,
+ .mem.type = DM_IO_KMEM,
+ .mem.ptr.addr = buf,
+ };
+ region = (struct dm_io_region) {
+ .bdev = wb->cache_dev->bdev,
+ .sector = (1 << 11) - 1,
+ .count = 1,
+ };
+ IO(dm_safe_io(&io_req, 1, ®ion, NULL, false));
+
+ mempool_free(buf, wb->buf_1_pool);
+}
+
+int recorder_proc(void *data)
+{
+ struct wb_device *wb = data;
+
+ unsigned long intvl;
+
+ while (!kthread_should_stop()) {
+ /* sec -> ms */
+ intvl = ACCESS_ONCE(wb->update_record_interval) * 1000;
+
+ if (!intvl) {
+ schedule_timeout_interruptible(msecs_to_jiffies(1000));
+ continue;
+ }
+
+ update_superblock_record(wb);
+ schedule_timeout_interruptible(msecs_to_jiffies(intvl));
+ }
+ return 0;
+}
+
+/*----------------------------------------------------------------*/
+
+int sync_proc(void *data)
+{
+ int r = 0;
+
+ struct wb_device *wb = data;
+ unsigned long intvl;
+
+ while (!kthread_should_stop()) {
+ /* sec -> ms */
+ intvl = ACCESS_ONCE(wb->sync_interval) * 1000;
+
+ if (!intvl) {
+ schedule_timeout_interruptible(msecs_to_jiffies(1000));
+ continue;
+ }
+
+ flush_current_buffer(wb);
+ IO(blkdev_issue_flush(wb->cache_dev->bdev, GFP_NOIO, NULL));
+ schedule_timeout_interruptible(msecs_to_jiffies(intvl));
+ }
+ return 0;
+}
diff --git a/drivers/md/dm-writeboost-daemon.h b/drivers/md/dm-writeboost-daemon.h
new file mode 100644
index 0000000..7e913db
--- /dev/null
+++ b/drivers/md/dm-writeboost-daemon.h
@@ -0,0 +1,40 @@
+/*
+ * Copyright (C) 2012-2014 Akira Hayakawa <ruby.wktk at gmail.com>
+ *
+ * This file is released under the GPL.
+ */
+
+#ifndef DM_WRITEBOOST_DAEMON_H
+#define DM_WRITEBOOST_DAEMON_H
+
+/*----------------------------------------------------------------*/
+
+void flush_proc(struct work_struct *);
+void wait_for_flushing(struct wb_device *, u64 id);
+
+/*----------------------------------------------------------------*/
+
+void queue_barrier_io(struct wb_device *, struct bio *);
+void barrier_deadline_proc(unsigned long data);
+void flush_barrier_ios(struct work_struct *);
+
+/*----------------------------------------------------------------*/
+
+int migrate_proc(void *);
+void wait_for_migration(struct wb_device *, u64 id);
+
+/*----------------------------------------------------------------*/
+
+int modulator_proc(void *);
+
+/*----------------------------------------------------------------*/
+
+int sync_proc(void *);
+
+/*----------------------------------------------------------------*/
+
+int recorder_proc(void *);
+
+/*----------------------------------------------------------------*/
+
+#endif
diff --git a/drivers/md/dm-writeboost-metadata.c b/drivers/md/dm-writeboost-metadata.c
new file mode 100644
index 0000000..54a94f5
--- /dev/null
+++ b/drivers/md/dm-writeboost-metadata.c
@@ -0,0 +1,1352 @@
+/*
+ * Copyright (C) 2012-2014 Akira Hayakawa <ruby.wktk at gmail.com>
+ *
+ * This file is released under the GPL.
+ */
+
+#include "dm-writeboost.h"
+#include "dm-writeboost-metadata.h"
+#include "dm-writeboost-daemon.h"
+
+#include <linux/crc32c.h>
+
+/*----------------------------------------------------------------*/
+
+struct part {
+ void *memory;
+};
+
+struct large_array {
+ struct part *parts;
+ u64 nr_elems;
+ u32 elemsize;
+};
+
+#define ALLOC_SIZE (1 << 16)
+static u32 nr_elems_in_part(struct large_array *arr)
+{
+ return div_u64(ALLOC_SIZE, arr->elemsize);
+};
+
+static u64 nr_parts(struct large_array *arr)
+{
+ u64 a = arr->nr_elems;
+ u32 b = nr_elems_in_part(arr);
+ return div_u64(a + b - 1, b);
+}
+
+static struct large_array *large_array_alloc(u32 elemsize, u64 nr_elems)
+{
+ u64 i;
+
+ struct large_array *arr = kmalloc(sizeof(*arr), GFP_KERNEL);
+ if (!arr) {
+ WBERR("failed to allocate arr");
+ return NULL;
+ }
+
+ arr->elemsize = elemsize;
+ arr->nr_elems = nr_elems;
+ arr->parts = kmalloc(sizeof(struct part) * nr_parts(arr), GFP_KERNEL);
+ if (!arr->parts) {
+ WBERR("failed to allocate parts");
+ goto bad_alloc_parts;
+ }
+
+ for (i = 0; i < nr_parts(arr); i++) {
+ struct part *part = arr->parts + i;
+ part->memory = kmalloc(ALLOC_SIZE, GFP_KERNEL);
+ if (!part->memory) {
+ u8 j;
+
+ WBERR("failed to allocate part memory");
+ for (j = 0; j < i; j++) {
+ part = arr->parts + j;
+ kfree(part->memory);
+ }
+ goto bad_alloc_parts_memory;
+ }
+ }
+ return arr;
+
+bad_alloc_parts_memory:
+ kfree(arr->parts);
+bad_alloc_parts:
+ kfree(arr);
+ return NULL;
+}
+
+static void large_array_free(struct large_array *arr)
+{
+ size_t i;
+ for (i = 0; i < nr_parts(arr); i++) {
+ struct part *part = arr->parts + i;
+ kfree(part->memory);
+ }
+ kfree(arr->parts);
+ kfree(arr);
+}
+
+static void *large_array_at(struct large_array *arr, u64 i)
+{
+ u32 n = nr_elems_in_part(arr);
+ u32 k;
+ u64 j = div_u64_rem(i, n, &k);
+ struct part *part = arr->parts + j;
+ return part->memory + (arr->elemsize * k);
+}
+
+/*----------------------------------------------------------------*/
+
+/*
+ * Get the in-core metablock of the given index.
+ */
+static struct metablock *mb_at(struct wb_device *wb, u32 idx)
+{
+ u32 idx_inseg;
+ u32 seg_idx = div_u64_rem(idx, wb->nr_caches_inseg, &idx_inseg);
+ struct segment_header *seg =
+ large_array_at(wb->segment_header_array, seg_idx);
+ return seg->mb_array + idx_inseg;
+}
+
+static void mb_array_empty_init(struct wb_device *wb)
+{
+ u32 i;
+ for (i = 0; i < wb->nr_caches; i++) {
+ struct metablock *mb = mb_at(wb, i);
+ INIT_HLIST_NODE(&mb->ht_list);
+
+ mb->idx = i;
+ mb->dirty_bits = 0;
+ }
+}
+
+/*
+ * Calc the starting sector of the k-th segment
+ */
+static sector_t calc_segment_header_start(struct wb_device *wb, u32 k)
+{
+ return (1 << 11) + (1 << wb->segment_size_order) * k;
+}
+
+static u32 calc_nr_segments(struct dm_dev *dev, struct wb_device *wb)
+{
+ sector_t devsize = dm_devsize(dev);
+ return div_u64(devsize - (1 << 11), 1 << wb->segment_size_order);
+}
+
+/*
+ * Get the relative index in a segment of the mb_idx-th metablock
+ */
+u32 mb_idx_inseg(struct wb_device *wb, u32 mb_idx)
+{
+ u32 tmp32;
+ div_u64_rem(mb_idx, wb->nr_caches_inseg, &tmp32);
+ return tmp32;
+}
+
+/*
+ * Calc the starting sector of the mb_idx-th cache block
+ */
+sector_t calc_mb_start_sector(struct wb_device *wb, struct segment_header *seg, u32 mb_idx)
+{
+ return seg->start_sector + ((1 + mb_idx_inseg(wb, mb_idx)) << 3);
+}
+
+/*
+ * Get the segment that contains the passed mb
+ */
+struct segment_header *mb_to_seg(struct wb_device *wb, struct metablock *mb)
+{
+ struct segment_header *seg;
+ seg = ((void *) mb)
+ - mb_idx_inseg(wb, mb->idx) * sizeof(struct metablock)
+ - sizeof(struct segment_header);
+ return seg;
+}
+
+bool is_on_buffer(struct wb_device *wb, u32 mb_idx)
+{
+ u32 start = wb->current_seg->start_idx;
+ if (mb_idx < start)
+ return false;
+
+ if (mb_idx >= (start + wb->nr_caches_inseg))
+ return false;
+
+ return true;
+}
+
+static u32 segment_id_to_idx(struct wb_device *wb, u64 id)
+{
+ u32 idx;
+ div_u64_rem(id - 1, wb->nr_segments, &idx);
+ return idx;
+}
+
+static struct segment_header *segment_at(struct wb_device *wb, u32 k)
+{
+ return large_array_at(wb->segment_header_array, k);
+}
+
+/*
+ * Get the segment from the segment id.
+ * The Index of the segment is calculated from the segment id.
+ */
+struct segment_header *
+get_segment_header_by_id(struct wb_device *wb, u64 id)
+{
+ return segment_at(wb, segment_id_to_idx(wb, id));
+}
+
+/*----------------------------------------------------------------*/
+
+static int __must_check init_segment_header_array(struct wb_device *wb)
+{
+ u32 segment_idx;
+
+ wb->segment_header_array = large_array_alloc(
+ sizeof(struct segment_header) +
+ sizeof(struct metablock) * wb->nr_caches_inseg,
+ wb->nr_segments);
+ if (!wb->segment_header_array) {
+ WBERR("failed to allocate segment header array");
+ return -ENOMEM;
+ }
+
+ for (segment_idx = 0; segment_idx < wb->nr_segments; segment_idx++) {
+ struct segment_header *seg = large_array_at(wb->segment_header_array, segment_idx);
+
+ seg->id = 0;
+ seg->length = 0;
+ atomic_set(&seg->nr_inflight_ios, 0);
+
+ /*
+ * Const values
+ */
+ seg->start_idx = wb->nr_caches_inseg * segment_idx;
+ seg->start_sector = calc_segment_header_start(wb, segment_idx);
+ }
+
+ mb_array_empty_init(wb);
+
+ return 0;
+}
+
+static void free_segment_header_array(struct wb_device *wb)
+{
+ large_array_free(wb->segment_header_array);
+}
+
+/*----------------------------------------------------------------*/
+
+struct ht_head {
+ struct hlist_head ht_list;
+};
+
+/*
+ * Initialize the Hash Table.
+ */
+static int __must_check ht_empty_init(struct wb_device *wb)
+{
+ u32 idx;
+ size_t i, nr_heads;
+ struct large_array *arr;
+
+ wb->htsize = wb->nr_caches;
+ nr_heads = wb->htsize + 1;
+ arr = large_array_alloc(sizeof(struct ht_head), nr_heads);
+ if (!arr) {
+ WBERR("failed to allocate arr");
+ return -ENOMEM;
+ }
+
+ wb->htable = arr;
+
+ for (i = 0; i < nr_heads; i++) {
+ struct ht_head *hd = large_array_at(arr, i);
+ INIT_HLIST_HEAD(&hd->ht_list);
+ }
+
+ /*
+ * Our hashtable has one special bucket called null head.
+ * Orphan metablocks are linked to the null head.
+ */
+ wb->null_head = large_array_at(wb->htable, wb->htsize);
+
+ for (idx = 0; idx < wb->nr_caches; idx++) {
+ struct metablock *mb = mb_at(wb, idx);
+ hlist_add_head(&mb->ht_list, &wb->null_head->ht_list);
+ }
+
+ return 0;
+}
+
+static void free_ht(struct wb_device *wb)
+{
+ large_array_free(wb->htable);
+}
+
+struct ht_head *ht_get_head(struct wb_device *wb, struct lookup_key *key)
+{
+ u32 idx;
+ div_u64_rem(key->sector, wb->htsize, &idx);
+ return large_array_at(wb->htable, idx);
+}
+
+static bool mb_hit(struct metablock *mb, struct lookup_key *key)
+{
+ return mb->sector == key->sector;
+}
+
+/*
+ * Remove the metablock from the hashtable
+ * and link the orphan to the null head.
+ */
+void ht_del(struct wb_device *wb, struct metablock *mb)
+{
+ struct ht_head *null_head;
+
+ hlist_del(&mb->ht_list);
+
+ null_head = wb->null_head;
+ hlist_add_head(&mb->ht_list, &null_head->ht_list);
+}
+
+void ht_register(struct wb_device *wb, struct ht_head *head,
+ struct metablock *mb, struct lookup_key *key)
+{
+ hlist_del(&mb->ht_list);
+ hlist_add_head(&mb->ht_list, &head->ht_list);
+
+ mb->sector = key->sector;
+};
+
+struct metablock *ht_lookup(struct wb_device *wb, struct ht_head *head,
+ struct lookup_key *key)
+{
+ struct metablock *mb, *found = NULL;
+ hlist_for_each_entry(mb, &head->ht_list, ht_list) {
+ if (mb_hit(mb, key)) {
+ found = mb;
+ break;
+ }
+ }
+ return found;
+}
+
+/*
+ * Remove all the metablock in the segment from the lookup table.
+ */
+void discard_caches_inseg(struct wb_device *wb, struct segment_header *seg)
+{
+ u8 i;
+ for (i = 0; i < wb->nr_caches_inseg; i++) {
+ struct metablock *mb = seg->mb_array + i;
+ ht_del(wb, mb);
+ }
+}
+
+/*----------------------------------------------------------------*/
+
+static int read_superblock_header(struct superblock_header_device *sup,
+ struct wb_device *wb)
+{
+ int r = 0;
+ struct dm_io_request io_req_sup;
+ struct dm_io_region region_sup;
+
+ void *buf = kmalloc(1 << SECTOR_SHIFT, GFP_KERNEL);
+ if (!buf)
+ return -ENOMEM;
+
+ io_req_sup = (struct dm_io_request) {
+ .client = wb_io_client,
+ .bi_rw = READ,
+ .notify.fn = NULL,
+ .mem.type = DM_IO_KMEM,
+ .mem.ptr.addr = buf,
+ };
+ region_sup = (struct dm_io_region) {
+ .bdev = wb->cache_dev->bdev,
+ .sector = 0,
+ .count = 1,
+ };
+ r = dm_safe_io(&io_req_sup, 1, ®ion_sup, NULL, false);
+ if (r) {
+ WBERR("I/O failed");
+ goto bad_io;
+ }
+
+ memcpy(sup, buf, sizeof(*sup));
+
+bad_io:
+ kfree(buf);
+ return r;
+}
+
+/*
+ * Check if the cache device is already formatted.
+ * Returns 0 iff this routine runs without failure.
+ */
+static int __must_check
+audit_cache_device(struct wb_device *wb, bool *need_format, bool *allow_format)
+{
+ int r = 0;
+ struct superblock_header_device sup;
+ r = read_superblock_header(&sup, wb);
+ if (r) {
+ WBERR("failed to read superblock header");
+ return r;
+ }
+
+ *need_format = true;
+ *allow_format = false;
+
+ if (le32_to_cpu(sup.magic) != WB_MAGIC) {
+ *allow_format = true;
+ WBERR("superblock header: magic number invalid");
+ return 0;
+ }
+
+ if (sup.segment_size_order != wb->segment_size_order) {
+ WBERR("superblock header: segment order not same %u != %u",
+ sup.segment_size_order, wb->segment_size_order);
+ } else {
+ *need_format = false;
+ }
+
+ return r;
+}
+
+static int format_superblock_header(struct wb_device *wb)
+{
+ int r = 0;
+
+ struct dm_io_request io_req_sup;
+ struct dm_io_region region_sup;
+
+ struct superblock_header_device sup = {
+ .magic = cpu_to_le32(WB_MAGIC),
+ .segment_size_order = wb->segment_size_order,
+ };
+
+ void *buf = kzalloc(1 << SECTOR_SHIFT, GFP_KERNEL);
+ if (!buf)
+ return -ENOMEM;
+
+ memcpy(buf, &sup, sizeof(sup));
+
+ io_req_sup = (struct dm_io_request) {
+ .client = wb_io_client,
+ .bi_rw = WRITE_FUA,
+ .notify.fn = NULL,
+ .mem.type = DM_IO_KMEM,
+ .mem.ptr.addr = buf,
+ };
+ region_sup = (struct dm_io_region) {
+ .bdev = wb->cache_dev->bdev,
+ .sector = 0,
+ .count = 1,
+ };
+ r = dm_safe_io(&io_req_sup, 1, ®ion_sup, NULL, false);
+ if (r) {
+ WBERR("I/O failed");
+ goto bad_io;
+ }
+
+bad_io:
+ kfree(buf);
+ return 0;
+}
+
+struct format_segmd_context {
+ int err;
+ atomic64_t count;
+};
+
+static void format_segmd_endio(unsigned long error, void *__context)
+{
+ struct format_segmd_context *context = __context;
+ if (error)
+ context->err = 1;
+ atomic64_dec(&context->count);
+}
+
+static int zeroing_full_superblock(struct wb_device *wb)
+{
+ int r = 0;
+ struct dm_dev *dev = wb->cache_dev;
+
+ struct dm_io_request io_req_sup;
+ struct dm_io_region region_sup;
+
+ void *buf = kzalloc(1 << 20, GFP_KERNEL);
+ if (!buf)
+ return -ENOMEM;
+
+ io_req_sup = (struct dm_io_request) {
+ .client = wb_io_client,
+ .bi_rw = WRITE_FUA,
+ .notify.fn = NULL,
+ .mem.type = DM_IO_KMEM,
+ .mem.ptr.addr = buf,
+ };
+ region_sup = (struct dm_io_region) {
+ .bdev = dev->bdev,
+ .sector = 0,
+ .count = (1 << 11),
+ };
+ r = dm_safe_io(&io_req_sup, 1, ®ion_sup, NULL, false);
+ if (r) {
+ WBERR("I/O failed");
+ goto bad_io;
+ }
+
+bad_io:
+ kfree(buf);
+ return r;
+}
+
+static int format_all_segment_headers(struct wb_device *wb)
+{
+ int r = 0;
+ struct dm_dev *dev = wb->cache_dev;
+ u32 i, nr_segments = calc_nr_segments(dev, wb);
+
+ struct format_segmd_context context;
+
+ void *buf = kzalloc(1 << 12, GFP_KERNEL);
+ if (!buf)
+ return -ENOMEM;
+
+ atomic64_set(&context.count, nr_segments);
+ context.err = 0;
+
+ /*
+ * Submit all the writes asynchronously.
+ */
+ for (i = 0; i < nr_segments; i++) {
+ struct dm_io_request io_req_seg = {
+ .client = wb_io_client,
+ .bi_rw = WRITE,
+ .notify.fn = format_segmd_endio,
+ .notify.context = &context,
+ .mem.type = DM_IO_KMEM,
+ .mem.ptr.addr = buf,
+ };
+ struct dm_io_region region_seg = {
+ .bdev = dev->bdev,
+ .sector = calc_segment_header_start(wb, i),
+ .count = (1 << 3),
+ };
+ r = dm_safe_io(&io_req_seg, 1, ®ion_seg, NULL, false);
+ if (r) {
+ WBERR("I/O failed");
+ break;
+ }
+ }
+ kfree(buf);
+
+ if (r)
+ return r;
+
+ /*
+ * Wait for all the writes complete.
+ */
+ while (atomic64_read(&context.count))
+ schedule_timeout_interruptible(msecs_to_jiffies(100));
+
+ if (context.err) {
+ WBERR("I/O failed at last");
+ return -EIO;
+ }
+
+ return r;
+}
+
+/*
+ * Format superblock header and
+ * all the segment headers in a cache device
+ */
+static int __must_check format_cache_device(struct wb_device *wb)
+{
+ int r = 0;
+ struct dm_dev *dev = wb->cache_dev;
+ r = zeroing_full_superblock(wb);
+ if (r)
+ return r;
+ r = format_superblock_header(wb); /* first 512B */
+ if (r)
+ return r;
+ r = format_all_segment_headers(wb);
+ if (r)
+ return r;
+ r = blkdev_issue_flush(dev->bdev, GFP_KERNEL, NULL);
+ return r;
+}
+
+/*
+ * First check if the superblock and the passed arguments
+ * are consistent and re-format the cache structure if they are not.
+ * If you want to re-format the cache device you must zeroed out
+ * the first one sector of the device.
+ *
+ * After this, the segment_size_order is fixed.
+ */
+static int might_format_cache_device(struct wb_device *wb)
+{
+ int r = 0;
+
+ bool need_format, allow_format;
+ r = audit_cache_device(wb, &need_format, &allow_format);
+ if (r) {
+ WBERR("failed to audit cache device");
+ return r;
+ }
+
+ if (need_format) {
+ if (allow_format) {
+ r = format_cache_device(wb);
+ if (r) {
+ WBERR("failed to format cache device");
+ return r;
+ }
+ } else {
+ r = -EINVAL;
+ WBERR("cache device not allowed to format");
+ return r;
+ }
+ }
+
+ return r;
+}
+
+/*----------------------------------------------------------------*/
+
+static int __must_check
+read_superblock_record(struct superblock_record_device *record,
+ struct wb_device *wb)
+{
+ int r = 0;
+ struct dm_io_request io_req;
+ struct dm_io_region region;
+
+ void *buf = kmalloc(1 << SECTOR_SHIFT, GFP_KERNEL);
+ if (!buf) {
+ WBERR();
+ return -ENOMEM;
+ }
+
+ io_req = (struct dm_io_request) {
+ .client = wb_io_client,
+ .bi_rw = READ,
+ .notify.fn = NULL,
+ .mem.type = DM_IO_KMEM,
+ .mem.ptr.addr = buf,
+ };
+ region = (struct dm_io_region) {
+ .bdev = wb->cache_dev->bdev,
+ .sector = (1 << 11) - 1,
+ .count = 1,
+ };
+ r = dm_safe_io(&io_req, 1, ®ion, NULL, false);
+ if (r) {
+ WBERR("I/O failed");
+ goto bad_io;
+ }
+
+ memcpy(record, buf, sizeof(*record));
+
+bad_io:
+ kfree(buf);
+ return r;
+}
+
+/*
+ * Read whole segment on the cache device to a pre-allocated buffer.
+ */
+static int __must_check
+read_whole_segment(void *buf, struct wb_device *wb, struct segment_header *seg)
+{
+ struct dm_io_request io_req = {
+ .client = wb_io_client,
+ .bi_rw = READ,
+ .notify.fn = NULL,
+ .mem.type = DM_IO_KMEM,
+ .mem.ptr.addr = buf,
+ };
+ struct dm_io_region region = {
+ .bdev = wb->cache_dev->bdev,
+ .sector = seg->start_sector,
+ .count = 1 << wb->segment_size_order,
+ };
+ return dm_safe_io(&io_req, 1, ®ion, NULL, false);
+}
+
+/*
+ * We make a checksum of a segment from the valid data
+ * in a segment except the first 1 sector.
+ */
+static u32 calc_checksum(void *rambuffer, u8 length)
+{
+ unsigned int len = (4096 - 512) + 4096 * length;
+ return crc32c(WB_CKSUM_SEED, rambuffer + 512, len);
+}
+
+/*
+ * Complete metadata in a segment buffer.
+ */
+void prepare_segment_header_device(void *rambuffer,
+ struct wb_device *wb,
+ struct segment_header *src)
+{
+ struct segment_header_device *dest = rambuffer;
+ u32 i;
+
+ BUG_ON((src->length - 1) != mb_idx_inseg(wb, wb->cursor));
+
+ for (i = 0; i < src->length; i++) {
+ struct metablock *mb = src->mb_array + i;
+ struct metablock_device *mbdev = dest->mbarr + i;
+
+ mbdev->sector = cpu_to_le64(mb->sector);
+ mbdev->dirty_bits = mb->dirty_bits;
+ }
+
+ dest->id = cpu_to_le64(src->id);
+ dest->checksum = cpu_to_le32(calc_checksum(rambuffer, src->length));
+ dest->length = src->length;
+}
+
+static void
+apply_metablock_device(struct wb_device *wb, struct segment_header *seg,
+ struct segment_header_device *src, u8 i)
+{
+ struct lookup_key key;
+ struct ht_head *head;
+ struct metablock *found = NULL, *mb = seg->mb_array + i;
+ struct metablock_device *mbdev = src->mbarr + i;
+
+ mb->sector = le64_to_cpu(mbdev->sector);
+ mb->dirty_bits = mbdev->dirty_bits;
+
+ /*
+ * A metablock is usually dirty but the exception is that
+ * the one inserted by force flush.
+ * In that case, the first metablock in a segment is clean.
+ */
+ if (!mb->dirty_bits)
+ return;
+
+ key = (struct lookup_key) {
+ .sector = mb->sector,
+ };
+ head = ht_get_head(wb, &key);
+ found = ht_lookup(wb, head, &key);
+ if (found) {
+ bool overwrite_fullsize = (mb->dirty_bits == 255);
+ invalidate_previous_cache(wb, mb_to_seg(wb, found), found,
+ overwrite_fullsize);
+ }
+
+ inc_nr_dirty_caches(wb);
+ ht_register(wb, head, mb, &key);
+}
+
+/*
+ * Read the on-disk metadata of the segment and
+ * update the in-core cache metadata structure.
+ */
+static void
+apply_segment_header_device(struct wb_device *wb, struct segment_header *seg,
+ struct segment_header_device *src)
+{
+ u8 i;
+
+ seg->length = src->length;
+
+ for (i = 0; i < src->length; i++)
+ apply_metablock_device(wb, seg, src, i);
+}
+
+/*
+ * If the RAM buffer is non-volatile
+ * we first write back all the valid buffers on them.
+ * By doing this, the discussion on replay algorithm is closed
+ * in replaying logs on only cache device.
+ */
+static int writeback_non_volatile_buffers(struct wb_device *wb)
+{
+ return 0;
+}
+
+static int find_max_id(struct wb_device *wb, u64 *max_id)
+{
+ int r = 0;
+
+ void *rambuf = kmalloc(1 << (wb->segment_size_order + SECTOR_SHIFT),
+ GFP_KERNEL);
+ u32 k;
+
+ *max_id = 0;
+ for (k = 0; k < wb->nr_segments; k++) {
+ struct segment_header *seg = segment_at(wb, k);
+ struct segment_header_device *header;
+ r = read_whole_segment(rambuf, wb, seg);
+ if (r) {
+ kfree(rambuf);
+ return r;
+ }
+
+ header = rambuf;
+ if (le64_to_cpu(header->id) > *max_id)
+ *max_id = le64_to_cpu(header->id);
+ }
+ kfree(rambuf);
+ return r;
+}
+
+static int apply_valid_segments(struct wb_device *wb, u64 *max_id)
+{
+ int r = 0;
+ struct segment_header *seg;
+ struct segment_header_device *header;
+
+ void *rambuf = kmalloc(1 << (wb->segment_size_order + SECTOR_SHIFT),
+ GFP_KERNEL);
+
+ u32 i, start_idx = segment_id_to_idx(wb, *max_id + 1);
+ *max_id = 0;
+ for (i = start_idx; i < (start_idx + wb->nr_segments); i++) {
+ u32 checksum1, checksum2, k;
+ div_u64_rem(i, wb->nr_segments, &k);
+ seg = segment_at(wb, k);
+
+ r = read_whole_segment(rambuf, wb, seg);
+ if (r) {
+ kfree(rambuf);
+ return r;
+ }
+
+ header = rambuf;
+
+ if (!le64_to_cpu(header->id))
+ continue;
+
+ checksum1 = le32_to_cpu(header->checksum);
+ checksum2 = calc_checksum(rambuf, header->length);
+ if (checksum1 != checksum2) {
+ DMWARN("checksum inconsistent id:%llu checksum:%u != %u",
+ (long long unsigned int) le64_to_cpu(header->id),
+ checksum1, checksum2);
+ continue;
+ }
+
+ apply_segment_header_device(wb, seg, header);
+ *max_id = le64_to_cpu(header->id);
+ }
+ kfree(rambuf);
+ return r;
+}
+
+static int infer_last_migrated_id(struct wb_device *wb)
+{
+ int r = 0;
+
+ u64 record_id;
+ struct superblock_record_device uninitialized_var(record);
+ r = read_superblock_record(&record, wb);
+ if (r)
+ return r;
+
+ atomic64_set(&wb->last_migrated_segment_id,
+ atomic64_read(&wb->last_flushed_segment_id) > wb->nr_segments ?
+ atomic64_read(&wb->last_flushed_segment_id) - wb->nr_segments : 0);
+
+ record_id = le64_to_cpu(record.last_migrated_segment_id);
+ if (record_id > atomic64_read(&wb->last_migrated_segment_id))
+ atomic64_set(&wb->last_migrated_segment_id, record_id);
+
+ return r;
+}
+
+/*
+ * Replay all the log on the cache device to reconstruct
+ * the in-memory metadata.
+ *
+ * Algorithm:
+ * 1. find the maxium id
+ * 2. start from the right. iterate all the log.
+ * 2. skip if id=0 or checkum invalid
+ * 2. apply otherwise.
+ *
+ * This algorithm is robust for floppy SSD that may write
+ * a segment partially or lose data on its buffer on power fault.
+ *
+ * Even if number of threads flush segments in parallel and
+ * some of them loses atomicity because of power fault
+ * this robust algorithm works.
+ */
+static int replay_log_on_cache(struct wb_device *wb)
+{
+ int r = 0;
+ u64 max_id;
+
+ r = find_max_id(wb, &max_id);
+ if (r) {
+ WBERR("failed to find max id");
+ return r;
+ }
+ r = apply_valid_segments(wb, &max_id);
+ if (r) {
+ WBERR("failed to apply valid segments");
+ return r;
+ }
+
+ /*
+ * Setup last_flushed_segment_id
+ */
+ atomic64_set(&wb->last_flushed_segment_id, max_id);
+
+ /*
+ * Setup last_migrated_segment_id
+ */
+ infer_last_migrated_id(wb);
+
+ return r;
+}
+
+/*
+ * Acquire and initialize the first segment header for our caching.
+ */
+static void prepare_first_seg(struct wb_device *wb)
+{
+ u64 init_segment_id = atomic64_read(&wb->last_flushed_segment_id) + 1;
+ acquire_new_seg(wb, init_segment_id);
+
+ /*
+ * We always keep the intergrity between cursor
+ * and seg->length.
+ */
+ wb->cursor = wb->current_seg->start_idx;
+ wb->current_seg->length = 1;
+}
+
+/*
+ * Recover all the cache state from the
+ * persistent devices (non-volatile RAM and SSD).
+ */
+static int __must_check recover_cache(struct wb_device *wb)
+{
+ int r = 0;
+
+ r = writeback_non_volatile_buffers(wb);
+ if (r) {
+ WBERR("failed to write back all the persistent data on non-volatile RAM");
+ return r;
+ }
+
+ r = replay_log_on_cache(wb);
+ if (r) {
+ WBERR("failed to replay log");
+ return r;
+ }
+
+ prepare_first_seg(wb);
+ return 0;
+}
+
+/*----------------------------------------------------------------*/
+
+static int __must_check init_rambuf_pool(struct wb_device *wb)
+{
+ size_t i;
+ sector_t alloc_sz = 1 << wb->segment_size_order;
+ u32 nr = div_u64(wb->rambuf_pool_amount * 2, alloc_sz);
+
+ if (!nr)
+ return -EINVAL;
+
+ wb->nr_rambuf_pool = nr;
+ wb->rambuf_pool = kmalloc(sizeof(struct rambuffer) * nr,
+ GFP_KERNEL);
+ if (!wb->rambuf_pool)
+ return -ENOMEM;
+
+ for (i = 0; i < wb->nr_rambuf_pool; i++) {
+ size_t j;
+ struct rambuffer *rambuf = wb->rambuf_pool + i;
+
+ rambuf->data = kmalloc(alloc_sz << SECTOR_SHIFT, GFP_KERNEL);
+ if (!rambuf->data) {
+ WBERR("failed to allocate rambuf data");
+ for (j = 0; j < i; j++) {
+ rambuf = wb->rambuf_pool + j;
+ kfree(rambuf->data);
+ }
+ kfree(wb->rambuf_pool);
+ return -ENOMEM;
+ }
+ }
+
+ return 0;
+}
+
+static void free_rambuf_pool(struct wb_device *wb)
+{
+ size_t i;
+ for (i = 0; i < wb->nr_rambuf_pool; i++) {
+ struct rambuffer *rambuf = wb->rambuf_pool + i;
+ kfree(rambuf->data);
+ }
+ kfree(wb->rambuf_pool);
+}
+
+/*----------------------------------------------------------------*/
+
+/*
+ * Try to allocate new migration buffer by the nr_batch size.
+ * On success, it frees the old buffer.
+ *
+ * Bad User may set # of batches that can hardly allocate.
+ * This function is robust in that case.
+ */
+int try_alloc_migration_buffer(struct wb_device *wb, size_t nr_batch)
+{
+ int r = 0;
+
+ struct segment_header **emigrates;
+ void *buf;
+ void *snapshot;
+
+ emigrates = kmalloc(nr_batch * sizeof(struct segment_header *), GFP_KERNEL);
+ if (!emigrates) {
+ WBERR("failed to allocate emigrates");
+ r = -ENOMEM;
+ return r;
+ }
+
+ buf = vmalloc(nr_batch * (wb->nr_caches_inseg << 12));
+ if (!buf) {
+ WBERR("failed to allocate migration buffer");
+ r = -ENOMEM;
+ goto bad_alloc_buffer;
+ }
+
+ snapshot = kmalloc(nr_batch * wb->nr_caches_inseg, GFP_KERNEL);
+ if (!snapshot) {
+ WBERR("failed to allocate dirty snapshot");
+ r = -ENOMEM;
+ goto bad_alloc_snapshot;
+ }
+
+ /*
+ * Free old buffers
+ */
+ kfree(wb->emigrates); /* kfree(NULL) is safe */
+ if (wb->migrate_buffer)
+ vfree(wb->migrate_buffer);
+ kfree(wb->dirtiness_snapshot);
+
+ /*
+ * Swap by new values
+ */
+ wb->emigrates = emigrates;
+ wb->migrate_buffer = buf;
+ wb->dirtiness_snapshot = snapshot;
+ wb->nr_cur_batched_migration = nr_batch;
+
+ return r;
+
+bad_alloc_buffer:
+ kfree(wb->emigrates);
+bad_alloc_snapshot:
+ vfree(wb->migrate_buffer);
+
+ return r;
+}
+
+static void free_migration_buffer(struct wb_device *wb)
+{
+ kfree(wb->emigrates);
+ vfree(wb->migrate_buffer);
+ kfree(wb->dirtiness_snapshot);
+}
+
+/*----------------------------------------------------------------*/
+
+#define CREATE_DAEMON(name) \
+ do { \
+ wb->name##_daemon = kthread_create( \
+ name##_proc, wb, #name "_daemon"); \
+ if (IS_ERR(wb->name##_daemon)) { \
+ r = PTR_ERR(wb->name##_daemon); \
+ wb->name##_daemon = NULL; \
+ WBERR("couldn't spawn " #name " daemon"); \
+ goto bad_##name##_daemon; \
+ } \
+ wake_up_process(wb->name##_daemon); \
+ } while (0)
+
+/*
+ * Setup the core info relavant to the cache format or geometry.
+ */
+static void setup_geom_info(struct wb_device *wb)
+{
+ wb->nr_segments = calc_nr_segments(wb->cache_dev, wb);
+ wb->nr_caches_inseg = (1 << (wb->segment_size_order - 3)) - 1;
+ wb->nr_caches = wb->nr_segments * wb->nr_caches_inseg;
+}
+
+/*
+ * Harmless init
+ * - allocate memory
+ * - setup the initial state of the objects
+ */
+static int harmless_init(struct wb_device *wb)
+{
+ int r = 0;
+
+ setup_geom_info(wb);
+
+ wb->buf_1_pool = mempool_create_kmalloc_pool(16, 1 << SECTOR_SHIFT);
+ if (!wb->buf_1_pool) {
+ r = -ENOMEM;
+ WBERR("failed to allocate 1 sector pool");
+ goto bad_buf_1_pool;
+ }
+ wb->buf_8_pool = mempool_create_kmalloc_pool(16, 8 << SECTOR_SHIFT);
+ if (!wb->buf_8_pool) {
+ r = -ENOMEM;
+ WBERR("failed to allocate 8 sector pool");
+ goto bad_buf_8_pool;
+ }
+
+ r = init_rambuf_pool(wb);
+ if (r) {
+ WBERR("failed to allocate rambuf pool");
+ goto bad_init_rambuf_pool;
+ }
+ wb->flush_job_pool = mempool_create_kmalloc_pool(
+ wb->nr_rambuf_pool, sizeof(struct flush_job));
+ if (!wb->flush_job_pool) {
+ r = -ENOMEM;
+ WBERR("failed to allocate flush job pool");
+ goto bad_flush_job_pool;
+ }
+
+ r = init_segment_header_array(wb);
+ if (r) {
+ WBERR("failed to allocate segment header array");
+ goto bad_alloc_segment_header_array;
+ }
+
+ r = ht_empty_init(wb);
+ if (r) {
+ WBERR("failed to allocate hashtable");
+ goto bad_alloc_ht;
+ }
+
+ return r;
+
+bad_alloc_ht:
+ free_segment_header_array(wb);
+bad_alloc_segment_header_array:
+ mempool_destroy(wb->flush_job_pool);
+bad_flush_job_pool:
+ free_rambuf_pool(wb);
+bad_init_rambuf_pool:
+ mempool_destroy(wb->buf_8_pool);
+bad_buf_8_pool:
+ mempool_destroy(wb->buf_1_pool);
+bad_buf_1_pool:
+
+ return r;
+}
+
+static void harmless_free(struct wb_device *wb)
+{
+ free_ht(wb);
+ free_segment_header_array(wb);
+ mempool_destroy(wb->flush_job_pool);
+ free_rambuf_pool(wb);
+ mempool_destroy(wb->buf_8_pool);
+ mempool_destroy(wb->buf_1_pool);
+}
+
+static int init_migrate_daemon(struct wb_device *wb)
+{
+ int r = 0;
+ size_t nr_batch;
+
+ atomic_set(&wb->migrate_fail_count, 0);
+ atomic_set(&wb->migrate_io_count, 0);
+
+ /*
+ * Default number of batched migration is 1MB / segment size.
+ * An ordinary HDD can afford at least 1MB/sec.
+ */
+ nr_batch = 1 << (11 - wb->segment_size_order);
+ wb->nr_max_batched_migration = nr_batch;
+ if (try_alloc_migration_buffer(wb, nr_batch))
+ return -ENOMEM;
+
+ init_waitqueue_head(&wb->migrate_wait_queue);
+ init_waitqueue_head(&wb->wait_drop_caches);
+ init_waitqueue_head(&wb->migrate_io_wait_queue);
+
+ wb->allow_migrate = false;
+ wb->urge_migrate = false;
+ CREATE_DAEMON(migrate);
+
+ return r;
+
+bad_migrate_daemon:
+ free_migration_buffer(wb);
+ return r;
+}
+
+static int init_flusher(struct wb_device *wb)
+{
+ int r = 0;
+ wb->flusher_wq = alloc_workqueue(
+ "%s", WQ_MEM_RECLAIM | WQ_SYSFS, 1, "wbflusher");
+ if (!wb->flusher_wq) {
+ WBERR("failed to allocate wbflusher");
+ return -ENOMEM;
+ }
+ init_waitqueue_head(&wb->flush_wait_queue);
+ return r;
+}
+
+static void init_barrier_deadline_work(struct wb_device *wb)
+{
+ wb->barrier_deadline_ms = 3;
+ setup_timer(&wb->barrier_deadline_timer,
+ barrier_deadline_proc, (unsigned long) wb);
+ bio_list_init(&wb->barrier_ios);
+ INIT_WORK(&wb->barrier_deadline_work, flush_barrier_ios);
+}
+
+static int init_migrate_modulator(struct wb_device *wb)
+{
+ int r = 0;
+ /*
+ * EMC's textbook on storage system teaches us
+ * storage should keep its load no more than 70%.
+ */
+ wb->migrate_threshold = 70;
+ wb->enable_migration_modulator = true;
+ CREATE_DAEMON(modulator);
+ return r;
+
+bad_modulator_daemon:
+ return r;
+}
+
+static int init_recorder_daemon(struct wb_device *wb)
+{
+ int r = 0;
+ wb->update_record_interval = 60;
+ CREATE_DAEMON(recorder);
+ return r;
+
+bad_recorder_daemon:
+ return r;
+}
+
+static int init_sync_daemon(struct wb_device *wb)
+{
+ int r = 0;
+ wb->sync_interval = 60;
+ CREATE_DAEMON(sync);
+ return r;
+
+bad_sync_daemon:
+ return r;
+}
+
+int __must_check resume_cache(struct wb_device *wb)
+{
+ int r = 0;
+
+ r = might_format_cache_device(wb);
+ if (r)
+ goto bad_might_format_cache;
+ r = harmless_init(wb);
+ if (r)
+ goto bad_harmless_init;
+ r = init_migrate_daemon(wb);
+ if (r) {
+ WBERR("failed to init migrate daemon");
+ goto bad_migrate_daemon;
+ }
+ r = recover_cache(wb);
+ if (r) {
+ WBERR("failed to recover cache metadata");
+ goto bad_recover;
+ }
+ r = init_flusher(wb);
+ if (r) {
+ WBERR("failed to init wbflusher");
+ goto bad_flusher;
+ }
+ init_barrier_deadline_work(wb);
+ r = init_migrate_modulator(wb);
+ if (r) {
+ WBERR("failed to init migrate modulator");
+ goto bad_migrate_modulator;
+ }
+ r = init_recorder_daemon(wb);
+ if (r) {
+ WBERR("failed to init superblock recorder");
+ goto bad_recorder_daemon;
+ }
+ r = init_sync_daemon(wb);
+ if (r) {
+ WBERR("failed to init sync daemon");
+ goto bad_sync_daemon;
+ }
+
+ return r;
+
+bad_sync_daemon:
+ kthread_stop(wb->recorder_daemon);
+bad_recorder_daemon:
+ kthread_stop(wb->modulator_daemon);
+bad_migrate_modulator:
+ cancel_work_sync(&wb->barrier_deadline_work);
+ destroy_workqueue(wb->flusher_wq);
+bad_flusher:
+bad_recover:
+ kthread_stop(wb->migrate_daemon);
+ free_migration_buffer(wb);
+bad_migrate_daemon:
+ harmless_free(wb);
+bad_harmless_init:
+bad_might_format_cache:
+
+ return r;
+}
+
+void free_cache(struct wb_device *wb)
+{
+ /*
+ * kthread_stop() wakes up the thread.
+ * We don't need to wake them up in our code.
+ */
+ kthread_stop(wb->sync_daemon);
+ kthread_stop(wb->recorder_daemon);
+ kthread_stop(wb->modulator_daemon);
+
+ cancel_work_sync(&wb->barrier_deadline_work);
+
+ destroy_workqueue(wb->flusher_wq);
+
+ kthread_stop(wb->migrate_daemon);
+ free_migration_buffer(wb);
+
+ harmless_free(wb);
+}
diff --git a/drivers/md/dm-writeboost-metadata.h b/drivers/md/dm-writeboost-metadata.h
new file mode 100644
index 0000000..860e4bf
--- /dev/null
+++ b/drivers/md/dm-writeboost-metadata.h
@@ -0,0 +1,51 @@
+/*
+ * Copyright (C) 2012-2014 Akira Hayakawa <ruby.wktk at gmail.com>
+ *
+ * This file is released under the GPL.
+ */
+
+#ifndef DM_WRITEBOOST_METADATA_H
+#define DM_WRITEBOOST_METADATA_H
+
+/*----------------------------------------------------------------*/
+
+struct segment_header *
+get_segment_header_by_id(struct wb_device *, u64 segment_id);
+sector_t calc_mb_start_sector(struct wb_device *, struct segment_header *,
+ u32 mb_idx);
+u32 mb_idx_inseg(struct wb_device *, u32 mb_idx);
+struct segment_header *mb_to_seg(struct wb_device *, struct metablock *);
+bool is_on_buffer(struct wb_device *, u32 mb_idx);
+
+/*----------------------------------------------------------------*/
+
+struct lookup_key {
+ sector_t sector;
+};
+
+struct ht_head;
+struct ht_head *ht_get_head(struct wb_device *, struct lookup_key *);
+struct metablock *ht_lookup(struct wb_device *,
+ struct ht_head *, struct lookup_key *);
+void ht_register(struct wb_device *, struct ht_head *,
+ struct metablock *, struct lookup_key *);
+void ht_del(struct wb_device *, struct metablock *);
+void discard_caches_inseg(struct wb_device *, struct segment_header *);
+
+/*----------------------------------------------------------------*/
+
+void prepare_segment_header_device(void *rambuffer, struct wb_device *,
+ struct segment_header *src);
+
+/*----------------------------------------------------------------*/
+
+int try_alloc_migration_buffer(struct wb_device *, size_t nr_batch);
+
+/*----------------------------------------------------------------*/
+
+int __must_check resume_cache(struct wb_device *);
+void free_cache(struct wb_device *);
+
+/*----------------------------------------------------------------*/
+
+#endif
diff --git a/drivers/md/dm-writeboost-target.c b/drivers/md/dm-writeboost-target.c
new file mode 100644
index 0000000..4cbf579
--- /dev/null
+++ b/drivers/md/dm-writeboost-target.c
@@ -0,0 +1,1258 @@
+/*
+ * Writeboost
+ * Log-structured Caching for Linux
+ *
+ * Copyright (C) 2012-2014 Akira Hayakawa <ruby.wktk at gmail.com>
+ *
+ * This file is released under the GPL.
+ */
+
+#include "dm-writeboost.h"
+#include "dm-writeboost-metadata.h"
+#include "dm-writeboost-daemon.h"
+
+/*----------------------------------------------------------------*/
+
+struct safe_io {
+ struct work_struct work;
+ int err;
+ unsigned long err_bits;
+ struct dm_io_request *io_req;
+ unsigned num_regions;
+ struct dm_io_region *regions;
+};
+
+static void safe_io_proc(struct work_struct *work)
+{
+ struct safe_io *io = container_of(work, struct safe_io, work);
+ io->err_bits = 0;
+ io->err = dm_io(io->io_req, io->num_regions, io->regions,
+ &io->err_bits);
+}
+
+int dm_safe_io_internal(struct wb_device *wb, struct dm_io_request *io_req,
+ unsigned num_regions, struct dm_io_region *regions,
+ unsigned long *err_bits, bool thread, const char *caller)
+{
+ int err = 0;
+
+ if (thread) {
+ struct safe_io io = {
+ .io_req = io_req,
+ .regions = regions,
+ .num_regions = num_regions,
+ };
+
+ INIT_WORK_ONSTACK(&io.work, safe_io_proc);
+
+ queue_work(safe_io_wq, &io.work);
+ flush_work(&io.work);
+
+ err = io.err;
+ if (err_bits)
+ *err_bits = io.err_bits;
+ } else {
+ err = dm_io(io_req, num_regions, regions, err_bits);
+ }
+
+ /*
+ * err_bits can be NULL.
+ */
+ if (err || (err_bits && *err_bits)) {
+ char buf[BDEVNAME_SIZE];
+ dev_t dev = regions->bdev->bd_dev;
+
+ unsigned long eb;
+ if (!err_bits)
+ eb = (~(unsigned long)0);
+ else
+ eb = *err_bits;
+
+ format_dev_t(buf, dev);
+ WBERR("%s() I/O error(%d), bits(%lu), dev(%s), sector(%llu), rw(%d)",
+ caller, err, eb,
+ buf, (unsigned long long) regions->sector, io_req->bi_rw);
+ }
+
+ return err;
+}
+
+sector_t dm_devsize(struct dm_dev *dev)
+{
+ return i_size_read(dev->bdev->bd_inode) >> SECTOR_SHIFT;
+}
+
+/*----------------------------------------------------------------*/
+
+static u8 count_dirty_caches_remained(struct segment_header *seg)
+{
+ u8 i, count = 0;
+ struct metablock *mb;
+ for (i = 0; i < seg->length; i++) {
+ mb = seg->mb_array + i;
+ if (mb->dirty_bits)
+ count++;
+ }
+ return count;
+}
+
+/*
+ * Prepare the kmalloc-ed RAM buffer for segment write.
+ *
+ * dm_io routine requires RAM buffer for its I/O buffer.
+ * Even if we uses non-volatile RAM we have to copy the
+ * data to the volatile buffer when we come to submit I/O.
+ */
+static void prepare_rambuffer(struct rambuffer *rambuf,
+ struct wb_device *wb,
+ struct segment_header *seg)
+{
+ prepare_segment_header_device(rambuf->data, wb, seg);
+}
+
+static void init_rambuffer(struct wb_device *wb)
+{
+ memset(wb->current_rambuf->data, 0, 1 << 12);
+}
+
+/*
+ * Acquire new RAM buffer for the new segment.
+ */
+static void acquire_new_rambuffer(struct wb_device *wb, u64 id)
+{
+ struct rambuffer *next_rambuf;
+ u32 tmp32;
+
+ wait_for_flushing(wb, SUB_ID(id, wb->nr_rambuf_pool));
+
+ div_u64_rem(id - 1, wb->nr_rambuf_pool, &tmp32);
+ next_rambuf = wb->rambuf_pool + tmp32;
+
+ wb->current_rambuf = next_rambuf;
+
+ init_rambuffer(wb);
+}
+
+/*
+ * Acquire the new segment and RAM buffer for the following writes.
+ * Gurantees all dirty caches in the segments are migrated and all metablocks
+ * in it are invalidated (linked to null head).
+ */
+void acquire_new_seg(struct wb_device *wb, u64 id)
+{
+ struct segment_header *new_seg = get_segment_header_by_id(wb, id);
+
+ /*
+ * We wait for all requests to the new segment is consumed.
+ * Mutex taken gurantees that no new I/O to this segment is coming in.
+ */
+ size_t rep = 0;
+ while (atomic_read(&new_seg->nr_inflight_ios)) {
+ rep++;
+ if (rep == 1000)
+ WBWARN("too long to process all requests");
+ schedule_timeout_interruptible(msecs_to_jiffies(1));
+ }
+ BUG_ON(count_dirty_caches_remained(new_seg));
+
+ wait_for_migration(wb, SUB_ID(id, wb->nr_segments));
+
+ discard_caches_inseg(wb, new_seg);
+
+ /*
+ * We must not set new id to the new segment before
+ * all wait_* events are done since they uses those id for waiting.
+ */
+ new_seg->id = id;
+ wb->current_seg = new_seg;
+
+ acquire_new_rambuffer(wb, id);
+}
+
+static void prepare_new_seg(struct wb_device *wb)
+{
+ u64 next_id = wb->current_seg->id + 1;
+ acquire_new_seg(wb, next_id);
+
+ /*
+ * Set the cursor to the last of the flushed segment.
+ */
+ wb->cursor = wb->current_seg->start_idx + (wb->nr_caches_inseg - 1);
+ wb->current_seg->length = 0;
+}
+
+static void
+copy_barrier_requests(struct flush_job *job, struct wb_device *wb)
+{
+ bio_list_init(&job->barrier_ios);
+ bio_list_merge(&job->barrier_ios, &wb->barrier_ios);
+ bio_list_init(&wb->barrier_ios);
+}
+
+static void init_flush_job(struct flush_job *job, struct wb_device *wb)
+{
+ job->wb = wb;
+ job->seg = wb->current_seg;
+ job->rambuf = wb->current_rambuf;
+
+ copy_barrier_requests(job, wb);
+}
+
+static void queue_flush_job(struct wb_device *wb)
+{
+ struct flush_job *job;
+ size_t rep = 0;
+
+ while (atomic_read(&wb->current_seg->nr_inflight_ios)) {
+ rep++;
+ if (rep == 1000)
+ WBWARN("too long to process all requests");
+ schedule_timeout_interruptible(msecs_to_jiffies(1));
+ }
+ prepare_rambuffer(wb->current_rambuf, wb, wb->current_seg);
+
+ job = mempool_alloc(wb->flush_job_pool, GFP_NOIO);
+ init_flush_job(job, wb);
+ INIT_WORK(&job->work, flush_proc);
+ queue_work(wb->flusher_wq, &job->work);
+}
+
+static void queue_current_buffer(struct wb_device *wb)
+{
+ queue_flush_job(wb);
+ prepare_new_seg(wb);
+}
+
+/*
+ * Flush out all the transient data at a moment but _NOT_ persistently.
+ * Clean up the writes before termination is an example of the usecase.
+ */
+void flush_current_buffer(struct wb_device *wb)
+{
+ struct segment_header *old_seg;
+
+ mutex_lock(&wb->io_lock);
+ old_seg = wb->current_seg;
+
+ queue_current_buffer(wb);
+
+ wb->cursor = wb->current_seg->start_idx;
+ wb->current_seg->length = 1;
+ mutex_unlock(&wb->io_lock);
+
+ wait_for_flushing(wb, old_seg->id);
+}
+
+/*----------------------------------------------------------------*/
+
+static void bio_remap(struct bio *bio, struct dm_dev *dev, sector_t sector)
+{
+ bio->bi_bdev = dev->bdev;
+ bio->bi_sector = sector;
+}
+
+static u8 io_offset(struct bio *bio)
+{
+ u32 tmp32;
+ div_u64_rem(bio->bi_sector, 1 << 3, &tmp32);
+ return tmp32;
+}
+
+static sector_t io_count(struct bio *bio)
+{
+ return bio->bi_size >> SECTOR_SHIFT;
+}
+
+static bool io_fullsize(struct bio *bio)
+{
+ return io_count(bio) == (1 << 3);
+}
+
+/*
+ * We use 4KB alignment address of original request the for the lookup key.
+ */
+static sector_t calc_cache_alignment(sector_t bio_sector)
+{
+ return div_u64(bio_sector, 1 << 3) * (1 << 3);
+}
+
+/*----------------------------------------------------------------*/
+
+static void inc_stat(struct wb_device *wb,
+ int rw, bool found, bool on_buffer, bool fullsize)
+{
+ atomic64_t *v;
+
+ int i = 0;
+ if (rw)
+ i |= (1 << STAT_WRITE);
+ if (found)
+ i |= (1 << STAT_HIT);
+ if (on_buffer)
+ i |= (1 << STAT_ON_BUFFER);
+ if (fullsize)
+ i |= (1 << STAT_FULLSIZE);
+
+ v = &wb->stat[i];
+ atomic64_inc(v);
+}
+
+static void clear_stat(struct wb_device *wb)
+{
+ size_t i;
+ for (i = 0; i < STATLEN; i++) {
+ atomic64_t *v = &wb->stat[i];
+ atomic64_set(v, 0);
+ }
+}
+
+/*----------------------------------------------------------------*/
+
+void inc_nr_dirty_caches(struct wb_device *wb)
+{
+ BUG_ON(!wb);
+ atomic64_inc(&wb->nr_dirty_caches);
+}
+
+static void dec_nr_dirty_caches(struct wb_device *wb)
+{
+ BUG_ON(!wb);
+ if (atomic64_dec_and_test(&wb->nr_dirty_caches))
+ wake_up_interruptible(&wb->wait_drop_caches);
+}
+
+/*
+ * Increase the dirtiness of a metablock.
+ */
+static void taint_mb(struct wb_device *wb, struct segment_header *seg,
+ struct metablock *mb, struct bio *bio)
+{
+ unsigned long flags;
+
+ bool was_clean = false;
+
+ spin_lock_irqsave(&wb->lock, flags);
+ if (!mb->dirty_bits) {
+ seg->length++;
+ BUG_ON(seg->length > wb->nr_caches_inseg);
+ was_clean = true;
+ }
+ if (likely(io_fullsize(bio))) {
+ mb->dirty_bits = 255;
+ } else {
+ u8 i;
+ u8 acc_bits = 0;
+ for (i = io_offset(bio); i < (io_offset(bio) + io_count(bio)); i++)
+ acc_bits += (1 << i);
+
+ mb->dirty_bits |= acc_bits;
+ }
+ BUG_ON(!io_count(bio));
+ BUG_ON(!mb->dirty_bits);
+ spin_unlock_irqrestore(&wb->lock, flags);
+
+ if (was_clean)
+ inc_nr_dirty_caches(wb);
+}
+
+void cleanup_mb_if_dirty(struct wb_device *wb, struct segment_header *seg,
+ struct metablock *mb)
+{
+ unsigned long flags;
+
+ bool was_dirty = false;
+
+ spin_lock_irqsave(&wb->lock, flags);
+ if (mb->dirty_bits) {
+ mb->dirty_bits = 0;
+ was_dirty = true;
+ }
+ spin_unlock_irqrestore(&wb->lock, flags);
+
+ if (was_dirty)
+ dec_nr_dirty_caches(wb);
+}
+
+/*
+ * Read the dirtiness of a metablock at the moment.
+ *
+ * In fact, I don't know if we should have the read statement surrounded
+ * by spinlock. Why I do this is that I worry about reading the
+ * intermediate value (neither the value of before-write nor after-write).
+ * Intel CPU guarantees it but other CPU may not.
+ * If any other CPU guarantees it we can remove the spinlock held.
+ */
+u8 read_mb_dirtiness(struct wb_device *wb, struct segment_header *seg,
+ struct metablock *mb)
+{
+ unsigned long flags;
+ u8 val;
+
+ spin_lock_irqsave(&wb->lock, flags);
+ val = mb->dirty_bits;
+ spin_unlock_irqrestore(&wb->lock, flags);
+
+ return val;
+}
+
+/*
+ * Migrate the caches in a metablock on the SSD (after flushed).
+ * The caches on the SSD are considered to be persistent so we need to
+ * write them back with WRITE_FUA flag.
+ */
+static void migrate_mb(struct wb_device *wb, struct segment_header *seg,
+ struct metablock *mb, u8 dirty_bits, bool thread)
+{
+ int r = 0;
+
+ if (!dirty_bits)
+ return;
+
+ if (dirty_bits == 255) {
+ void *buf = mempool_alloc(wb->buf_8_pool, GFP_NOIO);
+ struct dm_io_request io_req_r, io_req_w;
+ struct dm_io_region region_r, region_w;
+
+ io_req_r = (struct dm_io_request) {
+ .client = wb_io_client,
+ .bi_rw = READ,
+ .notify.fn = NULL,
+ .mem.type = DM_IO_KMEM,
+ .mem.ptr.addr = buf,
+ };
+ region_r = (struct dm_io_region) {
+ .bdev = wb->cache_dev->bdev,
+ .sector = calc_mb_start_sector(wb, seg, mb->idx),
+ .count = (1 << 3),
+ };
+ IO(dm_safe_io(&io_req_r, 1, ®ion_r, NULL, thread));
+
+ io_req_w = (struct dm_io_request) {
+ .client = wb_io_client,
+ .bi_rw = WRITE_FUA,
+ .notify.fn = NULL,
+ .mem.type = DM_IO_KMEM,
+ .mem.ptr.addr = buf,
+ };
+ region_w = (struct dm_io_region) {
+ .bdev = wb->origin_dev->bdev,
+ .sector = mb->sector,
+ .count = (1 << 3),
+ };
+ IO(dm_safe_io(&io_req_w, 1, ®ion_w, NULL, thread));
+
+ mempool_free(buf, wb->buf_8_pool);
+ } else {
+ void *buf = mempool_alloc(wb->buf_1_pool, GFP_NOIO);
+ u8 i;
+ for (i = 0; i < 8; i++) {
+ struct dm_io_request io_req_r, io_req_w;
+ struct dm_io_region region_r, region_w;
+
+ bool bit_on = dirty_bits & (1 << i);
+ if (!bit_on)
+ continue;
+
+ io_req_r = (struct dm_io_request) {
+ .client = wb_io_client,
+ .bi_rw = READ,
+ .notify.fn = NULL,
+ .mem.type = DM_IO_KMEM,
+ .mem.ptr.addr = buf,
+ };
+ region_r = (struct dm_io_region) {
+ .bdev = wb->cache_dev->bdev,
+ .sector = calc_mb_start_sector(wb, seg, mb->idx) + i,
+ .count = 1,
+ };
+ IO(dm_safe_io(&io_req_r, 1, ®ion_r, NULL, thread));
+
+ io_req_w = (struct dm_io_request) {
+ .client = wb_io_client,
+ .bi_rw = WRITE_FUA,
+ .notify.fn = NULL,
+ .mem.type = DM_IO_KMEM,
+ .mem.ptr.addr = buf,
+ };
+ region_w = (struct dm_io_region) {
+ .bdev = wb->origin_dev->bdev,
+ .sector = mb->sector + i,
+ .count = 1,
+ };
+ IO(dm_safe_io(&io_req_w, 1, ®ion_w, NULL, thread));
+ }
+ mempool_free(buf, wb->buf_1_pool);
+ }
+}
+
+/*
+ * Migrate the caches on the RAM buffer.
+ * Calling this function is really rare so the code is not optimal.
+ *
+ * Since the caches are of either one of these two status
+ * - not flushed and thus not persistent (volatile buffer)
+ * - acked to barrier request before but it is also on the
+ * non-volatile buffer (non-volatile buffer)
+ * there is no reason to write them back with FUA flag.
+ */
+static void migrate_buffered_mb(struct wb_device *wb,
+ struct metablock *mb, u8 dirty_bits)
+{
+ int r = 0;
+
+ sector_t offset = ((mb_idx_inseg(wb, mb->idx) + 1) << 3);
+ void *buf = mempool_alloc(wb->buf_1_pool, GFP_NOIO);
+
+ u8 i;
+ for (i = 0; i < 8; i++) {
+ struct dm_io_request io_req;
+ struct dm_io_region region;
+ void *src;
+ sector_t dest;
+
+ bool bit_on = dirty_bits & (1 << i);
+ if (!bit_on)
+ continue;
+
+ src = wb->current_rambuf->data +
+ ((offset + i) << SECTOR_SHIFT);
+ memcpy(buf, src, 1 << SECTOR_SHIFT);
+
+ io_req = (struct dm_io_request) {
+ .client = wb_io_client,
+ .bi_rw = WRITE,
+ .notify.fn = NULL,
+ .mem.type = DM_IO_KMEM,
+ .mem.ptr.addr = buf,
+ };
+
+ dest = mb->sector + i;
+ region = (struct dm_io_region) {
+ .bdev = wb->origin_dev->bdev,
+ .sector = dest,
+ .count = 1,
+ };
+
+ IO(dm_safe_io(&io_req, 1, ®ion, NULL, true));
+ }
+ mempool_free(buf, wb->buf_1_pool);
+}
+
+void invalidate_previous_cache(struct wb_device *wb, struct segment_header *seg,
+ struct metablock *old_mb, bool overwrite_fullsize)
+{
+ u8 dirty_bits = read_mb_dirtiness(wb, seg, old_mb);
+
+ /*
+ * First clean up the previous cache and migrate the cache if needed.
+ */
+ bool needs_cleanup_prev_cache =
+ !overwrite_fullsize || !(dirty_bits == 255);
+
+ /*
+ * Migration works in background and may have cleaned up the metablock.
+ * If the metablock is clean we need not to migrate.
+ */
+ if (!dirty_bits)
+ needs_cleanup_prev_cache = false;
+
+ if (overwrite_fullsize)
+ needs_cleanup_prev_cache = false;
+
+ if (unlikely(needs_cleanup_prev_cache)) {
+ wait_for_flushing(wb, seg->id);
+ migrate_mb(wb, seg, old_mb, dirty_bits, true);
+ }
+
+ cleanup_mb_if_dirty(wb, seg, old_mb);
+
+ ht_del(wb, old_mb);
+}
+
+static void
+write_on_buffer(struct wb_device *wb, struct segment_header *seg,
+ struct metablock *mb, struct bio *bio)
+{
+ sector_t start_sector = ((mb_idx_inseg(wb, mb->idx) + 1) << 3) +
+ io_offset(bio);
+ size_t start_byte = start_sector << SECTOR_SHIFT;
+ void *data = bio_data(bio);
+
+ /*
+ * Write data block to the volatile RAM buffer.
+ */
+ memcpy(wb->current_rambuf->data + start_byte, data, bio->bi_size);
+}
+
+static void advance_cursor(struct wb_device *wb)
+{
+ u32 tmp32;
+ div_u64_rem(wb->cursor + 1, wb->nr_caches, &tmp32);
+ wb->cursor = tmp32;
+}
+
+struct per_bio_data {
+ void *ptr;
+};
+
+static int writeboost_map(struct dm_target *ti, struct bio *bio)
+{
+ struct wb_device *wb = ti->private;
+ struct dm_dev *origin_dev = wb->origin_dev;
+ int rw = bio_data_dir(bio);
+ struct lookup_key key = {
+ .sector = calc_cache_alignment(bio->bi_sector),
+ };
+ struct ht_head *head = ht_get_head(wb, &key);
+
+ struct segment_header *uninitialized_var(found_seg);
+ struct metablock *mb, *new_mb;
+
+ bool found,
+ on_buffer, /* is the metablock found on the RAM buffer? */
+ needs_queue_seg; /* need to queue the current seg? */
+
+ struct per_bio_data *map_context;
+ map_context = dm_per_bio_data(bio, ti->per_bio_data_size);
+ map_context->ptr = NULL;
+
+ DEAD(bio_endio(bio, -EIO); return DM_MAPIO_SUBMITTED);
+
+ /*
+ * We only discard sectors on only the backing store because
+ * blocks on cache device are unlikely to be discarded.
+ * Discarding blocks is likely to be operated long after writing;
+ * the block is likely to be migrated before that.
+ *
+ * Moreover, it is very hard to implement discarding cache blocks.
+ */
+ if (bio->bi_rw & REQ_DISCARD) {
+ bio_remap(bio, origin_dev, bio->bi_sector);
+ return DM_MAPIO_REMAPPED;
+ }
+
+ /*
+ * Defered ACK for flush requests
+ *
+ * In device-mapper, bio with REQ_FLUSH is guaranteed to have no data.
+ * So, we can simply defer it for lazy execution.
+ */
+ if (bio->bi_rw & REQ_FLUSH) {
+ BUG_ON(bio->bi_size);
+ queue_barrier_io(wb, bio);
+ return DM_MAPIO_SUBMITTED;
+ }
+
+ mutex_lock(&wb->io_lock);
+ mb = ht_lookup(wb, head, &key);
+ if (mb) {
+ found_seg = mb_to_seg(wb, mb);
+ atomic_inc(&found_seg->nr_inflight_ios);
+ }
+
+ found = (mb != NULL);
+ on_buffer = false;
+ if (found)
+ on_buffer = is_on_buffer(wb, mb->idx);
+
+ inc_stat(wb, rw, found, on_buffer, io_fullsize(bio));
+
+ /*
+ * (Locking)
+ * A cache data is placed either on RAM buffer or SSD if it was flushed.
+ * To ease the locking, we establish a simple rule for the dirtiness
+ * of a cache data.
+ *
+ * If the data is on the RAM buffer, the dirtiness (dirty_bits of metablock)
+ * only increases. The justification for this design is that the cache on the
+ * RAM buffer is seldom migrated.
+ * If the data is, on the other hand, on the SSD after flushed the dirtiness
+ * only decreases.
+ *
+ * This simple rule frees us from the dirtiness fluctuating thus simplies
+ * locking design.
+ */
+
+ if (!rw) {
+ u8 dirty_bits;
+
+ mutex_unlock(&wb->io_lock);
+
+ if (!found) {
+ bio_remap(bio, origin_dev, bio->bi_sector);
+ return DM_MAPIO_REMAPPED;
+ }
+
+ dirty_bits = read_mb_dirtiness(wb, found_seg, mb);
+ if (unlikely(on_buffer)) {
+ if (dirty_bits)
+ migrate_buffered_mb(wb, mb, dirty_bits);
+
+ atomic_dec(&found_seg->nr_inflight_ios);
+ bio_remap(bio, origin_dev, bio->bi_sector);
+ return DM_MAPIO_REMAPPED;
+ }
+
+ /*
+ * We must wait for the (maybe) queued segment to be flushed
+ * to the cache device.
+ * Without this, we read the wrong data from the cache device.
+ */
+ wait_for_flushing(wb, found_seg->id);
+
+ if (likely(dirty_bits == 255)) {
+ bio_remap(bio, wb->cache_dev,
+ calc_mb_start_sector(wb, found_seg, mb->idx) +
+ io_offset(bio));
+ map_context->ptr = found_seg;
+ } else {
+ migrate_mb(wb, found_seg, mb, dirty_bits, true);
+ cleanup_mb_if_dirty(wb, found_seg, mb);
+
+ atomic_dec(&found_seg->nr_inflight_ios);
+ bio_remap(bio, origin_dev, bio->bi_sector);
+ }
+ return DM_MAPIO_REMAPPED;
+ }
+
+ if (found) {
+ if (unlikely(on_buffer)) {
+ mutex_unlock(&wb->io_lock);
+ goto write_on_buffer;
+ } else {
+ invalidate_previous_cache(wb, found_seg, mb,
+ io_fullsize(bio));
+ atomic_dec(&found_seg->nr_inflight_ios);
+ goto write_not_found;
+ }
+ }
+
+write_not_found:
+ /*
+ * If wb->cursor is 254, 509, ...
+ * which is the last cache line in the segment.
+ * We must flush the current segment and get the new one.
+ */
+ needs_queue_seg = !mb_idx_inseg(wb, wb->cursor + 1);
+
+ if (needs_queue_seg)
+ queue_current_buffer(wb);
+
+ advance_cursor(wb);
+
+ new_mb = wb->current_seg->mb_array + mb_idx_inseg(wb, wb->cursor);
+ BUG_ON(new_mb->dirty_bits);
+ ht_register(wb, head, new_mb, &key);
+
+ atomic_inc(&wb->current_seg->nr_inflight_ios);
+ mutex_unlock(&wb->io_lock);
+
+ mb = new_mb;
+
+write_on_buffer:
+ taint_mb(wb, wb->current_seg, mb, bio);
+
+ write_on_buffer(wb, wb->current_seg, mb, bio);
+
+ atomic_dec(&wb->current_seg->nr_inflight_ios);
+
+ /*
+ * Deferred ACK for FUA request
+ *
+ * bio with REQ_FUA flag has data.
+ * So, we must run through the path for usual bio.
+ * And the data is now stored in the RAM buffer.
+ */
+ if (bio->bi_rw & REQ_FUA) {
+ queue_barrier_io(wb, bio);
+ return DM_MAPIO_SUBMITTED;
+ }
+
+ LIVE_DEAD(bio_endio(bio, 0),
+ bio_endio(bio, -EIO));
+
+ return DM_MAPIO_SUBMITTED;
+}
+
+static int writeboost_end_io(struct dm_target *ti, struct bio *bio, int error)
+{
+ struct segment_header *seg;
+ struct per_bio_data *map_context =
+ dm_per_bio_data(bio, ti->per_bio_data_size);
+
+ if (!map_context->ptr)
+ return 0;
+
+ seg = map_context->ptr;
+ atomic_dec(&seg->nr_inflight_ios);
+
+ return 0;
+}
+
+static int consume_essential_argv(struct wb_device *wb, struct dm_arg_set *as)
+{
+ int r = 0;
+ struct dm_target *ti = wb->ti;
+
+ static struct dm_arg _args[] = {
+ {0, 0, "invalid buffer type"},
+ };
+ unsigned tmp;
+
+ r = dm_read_arg(_args, as, &tmp, &ti->error);
+ if (r)
+ return r;
+ wb->type = tmp;
+
+ r = dm_get_device(ti, dm_shift_arg(as), dm_table_get_mode(ti->table),
+ &wb->origin_dev);
+ if (r) {
+ ti->error = "failed to get origin dev";
+ return r;
+ }
+
+ r = dm_get_device(ti, dm_shift_arg(as), dm_table_get_mode(ti->table),
+ &wb->cache_dev);
+ if (r) {
+ ti->error = "failed to get cache dev";
+ goto bad;
+ }
+
+ return r;
+
+bad:
+ dm_put_device(ti, wb->origin_dev);
+ return r;
+}
+
+#define consume_kv(name, nr) { \
+ if (!strcasecmp(key, #name)) { \
+ if (!argc) \
+ break; \
+ r = dm_read_arg(_args + (nr), as, &tmp, &ti->error); \
+ if (r) \
+ break; \
+ wb->name = tmp; \
+ } }
+
+static int consume_optional_argv(struct wb_device *wb, struct dm_arg_set *as)
+{
+ int r = 0;
+ struct dm_target *ti = wb->ti;
+
+ static struct dm_arg _args[] = {
+ {0, 4, "invalid optional argc"},
+ {4, 10, "invalid segment_size_order"},
+ {512, UINT_MAX, "invalid rambuf_pool_amount"},
+ };
+ unsigned tmp, argc = 0;
+
+ if (as->argc) {
+ r = dm_read_arg_group(_args, as, &argc, &ti->error);
+ if (r)
+ return r;
+ }
+
+ while (argc) {
+ const char *key = dm_shift_arg(as);
+ argc--;
+
+ r = -EINVAL;
+
+ consume_kv(segment_size_order, 1);
+ consume_kv(rambuf_pool_amount, 2);
+
+ if (!r) {
+ argc--;
+ } else {
+ ti->error = "invalid optional key";
+ break;
+ }
+ }
+
+ return r;
+}
+
+static int do_consume_tunable_argv(struct wb_device *wb,
+ struct dm_arg_set *as, unsigned argc)
+{
+ int r = 0;
+ struct dm_target *ti = wb->ti;
+
+ static struct dm_arg _args[] = {
+ {0, 1, "invalid allow_migrate"},
+ {0, 1, "invalid enable_migration_modulator"},
+ {1, 1000, "invalid barrier_deadline_ms"},
+ {1, 1000, "invalid nr_max_batched_migration"},
+ {0, 100, "invalid migrate_threshold"},
+ {0, 3600, "invalid update_record_interval"},
+ {0, 3600, "invalid sync_interval"},
+ };
+ unsigned tmp;
+
+ while (argc) {
+ const char *key = dm_shift_arg(as);
+ argc--;
+
+ r = -EINVAL;
+
+ consume_kv(allow_migrate, 0);
+ consume_kv(enable_migration_modulator, 1);
+ consume_kv(barrier_deadline_ms, 2);
+ consume_kv(nr_max_batched_migration, 3);
+ consume_kv(migrate_threshold, 4);
+ consume_kv(update_record_interval, 5);
+ consume_kv(sync_interval, 6);
+
+ if (!r) {
+ argc--;
+ } else {
+ ti->error = "invalid tunable key";
+ break;
+ }
+ }
+
+ return r;
+}
+
+static int consume_tunable_argv(struct wb_device *wb, struct dm_arg_set *as)
+{
+ int r = 0;
+ struct dm_target *ti = wb->ti;
+
+ static struct dm_arg _args[] = {
+ {0, 14, "invalid tunable argc"},
+ };
+ unsigned argc = 0;
+
+ if (as->argc) {
+ r = dm_read_arg_group(_args, as, &argc, &ti->error);
+ if (r)
+ return r;
+ /*
+ * tunables are emitted only if
+ * they were origianlly passed.
+ */
+ wb->should_emit_tunables = true;
+ }
+
+ return do_consume_tunable_argv(wb, as, argc);
+}
+
+static int init_core_struct(struct dm_target *ti)
+{
+ int r = 0;
+ struct wb_device *wb;
+
+ r = dm_set_target_max_io_len(ti, 1 << 3);
+ if (r) {
+ WBERR("failed to set max_io_len");
+ return r;
+ }
+
+ ti->flush_supported = true;
+ ti->num_flush_bios = 1;
+ ti->num_discard_bios = 1;
+ ti->discard_zeroes_data_unsupported = true;
+ ti->per_bio_data_size = sizeof(struct per_bio_data);
+
+ wb = kzalloc(sizeof(*wb), GFP_KERNEL);
+ if (!wb) {
+ WBERR("failed to allocate wb");
+ return -ENOMEM;
+ }
+ ti->private = wb;
+ wb->ti = ti;
+
+ mutex_init(&wb->io_lock);
+ spin_lock_init(&wb->lock);
+ atomic64_set(&wb->nr_dirty_caches, 0);
+ clear_bit(WB_DEAD, &wb->flags);
+ wb->should_emit_tunables = false;
+
+ return r;
+}
+
+/*
+ * Create a Writeboost device
+ *
+ * <type>
+ * <essential args>*
+ * <#optional args> <optional args>*
+ * <#tunable args> <tunable args>*
+ * optionals are tunables are unordered lists of k-v pair.
+ *
+ * See Documentation for detail.
+ */
+static int writeboost_ctr(struct dm_target *ti, unsigned int argc, char **argv)
+{
+ int r = 0;
+ struct wb_device *wb;
+
+ struct dm_arg_set as;
+ as.argc = argc;
+ as.argv = argv;
+
+ r = init_core_struct(ti);
+ if (r) {
+ ti->error = "failed to init core";
+ return r;
+ }
+ wb = ti->private;
+
+ r = consume_essential_argv(wb, &as);
+ if (r) {
+ ti->error = "failed to consume essential argv";
+ goto bad_essential_argv;
+ }
+
+ wb->segment_size_order = 7;
+ wb->rambuf_pool_amount = 2048;
+ r = consume_optional_argv(wb, &as);
+ if (r) {
+ ti->error = "failed to consume optional argv";
+ goto bad_optional_argv;
+ }
+
+ r = resume_cache(wb);
+ if (r) {
+ ti->error = "failed to resume cache";
+ goto bad_resume_cache;
+ }
+
+ r = consume_tunable_argv(wb, &as);
+ if (r) {
+ ti->error = "failed to consume tunable argv";
+ goto bad_tunable_argv;
+ }
+
+ clear_stat(wb);
+ atomic64_set(&wb->count_non_full_flushed, 0);
+
+ return r;
+
+bad_tunable_argv:
+ free_cache(wb);
+bad_resume_cache:
+bad_optional_argv:
+ dm_put_device(ti, wb->cache_dev);
+ dm_put_device(ti, wb->origin_dev);
+bad_essential_argv:
+ kfree(wb);
+
+ return r;
+}
+
+static void writeboost_dtr(struct dm_target *ti)
+{
+ struct wb_device *wb = ti->private;
+
+ free_cache(wb);
+
+ dm_put_device(ti, wb->cache_dev);
+ dm_put_device(ti, wb->origin_dev);
+
+ kfree(wb);
+
+ ti->private = NULL;
+}
+
+/*
+ * .postsuspend is called before .dtr.
+ * We flush out all the transient data and make them persistent.
+ */
+static void writeboost_postsuspend(struct dm_target *ti)
+{
+ int r = 0;
+ struct wb_device *wb = ti->private;
+
+ flush_current_buffer(wb);
+ IO(blkdev_issue_flush(wb->cache_dev->bdev, GFP_NOIO, NULL));
+}
+
+static int writeboost_message(struct dm_target *ti, unsigned argc, char **argv)
+{
+ struct wb_device *wb = ti->private;
+
+ struct dm_arg_set as;
+ as.argc = argc;
+ as.argv = argv;
+
+ if (!strcasecmp(argv[0], "clear_stat")) {
+ clear_stat(wb);
+ return 0;
+ }
+
+ if (!strcasecmp(argv[0], "drop_caches")) {
+ int r = 0;
+ wb->force_drop = true;
+ r = wait_event_interruptible(wb->wait_drop_caches,
+ !atomic64_read(&wb->nr_dirty_caches));
+ wb->force_drop = false;
+ return r;
+ }
+
+ return do_consume_tunable_argv(wb, &as, 2);
+}
+
+/*
+ * Since Writeboost is just a cache target and the cache block size is fixed
+ * to 4KB. There is no reason to count the cache device in device iteration.
+ */
+static int
+writeboost_iterate_devices(struct dm_target *ti,
+ iterate_devices_callout_fn fn, void *data)
+{
+ struct wb_device *wb = ti->private;
+ struct dm_dev *orig = wb->origin_dev;
+ sector_t start = 0;
+ sector_t len = dm_devsize(orig);
+ return fn(ti, orig, start, len, data);
+}
+
+static void
+writeboost_io_hints(struct dm_target *ti, struct queue_limits *limits)
+{
+ blk_limits_io_opt(limits, 4096);
+}
+
+static void emit_tunables(struct wb_device *wb, char *result, unsigned maxlen)
+{
+ ssize_t sz = 0;
+
+ DMEMIT(" %d", 14);
+ DMEMIT(" barrier_deadline_ms %lu",
+ wb->barrier_deadline_ms);
+ DMEMIT(" allow_migrate %d",
+ wb->allow_migrate ? 1 : 0);
+ DMEMIT(" enable_migration_modulator %d",
+ wb->enable_migration_modulator ? 1 : 0);
+ DMEMIT(" migrate_threshold %d",
+ wb->migrate_threshold);
+ DMEMIT(" nr_cur_batched_migration %u",
+ wb->nr_cur_batched_migration);
+ DMEMIT(" sync_interval %lu",
+ wb->sync_interval);
+ DMEMIT(" update_record_interval %lu",
+ wb->update_record_interval);
+}
+
+static void writeboost_status(struct dm_target *ti, status_type_t type,
+ unsigned flags, char *result, unsigned maxlen)
+{
+ ssize_t sz = 0;
+ char buf[BDEVNAME_SIZE];
+ struct wb_device *wb = ti->private;
+ size_t i;
+
+ switch (type) {
+ case STATUSTYPE_INFO:
+ DMEMIT("%u %u %llu %llu %llu %llu %llu",
+ (unsigned int)
+ wb->cursor,
+ (unsigned int)
+ wb->nr_caches,
+ (long long unsigned int)
+ wb->nr_segments,
+ (long long unsigned int)
+ wb->current_seg->id,
+ (long long unsigned int)
+ atomic64_read(&wb->last_flushed_segment_id),
+ (long long unsigned int)
+ atomic64_read(&wb->last_migrated_segment_id),
+ (long long unsigned int)
+ atomic64_read(&wb->nr_dirty_caches));
+
+ for (i = 0; i < STATLEN; i++) {
+ atomic64_t *v = &wb->stat[i];
+ DMEMIT(" %llu", (unsigned long long) atomic64_read(v));
+ }
+ DMEMIT(" %llu", (unsigned long long) atomic64_read(&wb->count_non_full_flushed));
+ emit_tunables(wb, result + sz, maxlen - sz);
+ break;
+
+ case STATUSTYPE_TABLE:
+ DMEMIT("%u", wb->type);
+ format_dev_t(buf, wb->origin_dev->bdev->bd_dev),
+ DMEMIT(" %s", buf);
+ format_dev_t(buf, wb->cache_dev->bdev->bd_dev),
+ DMEMIT(" %s", buf);
+ DMEMIT(" 4 segment_size_order %u rambuf_pool_amount %u",
+ wb->segment_size_order,
+ wb->rambuf_pool_amount);
+ if (wb->should_emit_tunables)
+ emit_tunables(wb, result + sz, maxlen - sz);
+ break;
+ }
+}
+
+static struct target_type writeboost_target = {
+ .name = "writeboost",
+ .version = {0, 1, 0},
+ .module = THIS_MODULE,
+ .map = writeboost_map,
+ .end_io = writeboost_end_io,
+ .ctr = writeboost_ctr,
+ .dtr = writeboost_dtr,
+ /*
+ * .merge is not implemented
+ * We split the passed I/O into 4KB cache block no matter
+ * how big the I/O is.
+ */
+ .postsuspend = writeboost_postsuspend,
+ .message = writeboost_message,
+ .status = writeboost_status,
+ .io_hints = writeboost_io_hints,
+ .iterate_devices = writeboost_iterate_devices,
+};
+
+struct dm_io_client *wb_io_client;
+struct workqueue_struct *safe_io_wq;
+static int __init writeboost_module_init(void)
+{
+ int r = 0;
+
+ r = dm_register_target(&writeboost_target);
+ if (r < 0) {
+ WBERR("failed to register target");
+ return r;
+ }
+
+ safe_io_wq = alloc_workqueue("wbsafeiowq",
+ WQ_NON_REENTRANT | WQ_MEM_RECLAIM, 0);
+ if (!safe_io_wq) {
+ WBERR("failed to allocate safe_io_wq");
+ r = -ENOMEM;
+ goto bad_wq;
+ }
+
+ wb_io_client = dm_io_client_create();
+ if (IS_ERR(wb_io_client)) {
+ WBERR("failed to allocate wb_io_client");
+ r = PTR_ERR(wb_io_client);
+ goto bad_io_client;
+ }
+
+ return r;
+
+bad_io_client:
+ destroy_workqueue(safe_io_wq);
+bad_wq:
+ dm_unregister_target(&writeboost_target);
+
+ return r;
+}
+
+static void __exit writeboost_module_exit(void)
+{
+ dm_io_client_destroy(wb_io_client);
+ destroy_workqueue(safe_io_wq);
+ dm_unregister_target(&writeboost_target);
+}
+
+module_init(writeboost_module_init);
+module_exit(writeboost_module_exit);
+
+MODULE_AUTHOR("Akira Hayakawa <ruby.wktk at gmail.com>");
+MODULE_DESCRIPTION(DM_NAME " writeboost target");
+MODULE_LICENSE("GPL");
diff --git a/drivers/md/dm-writeboost.h b/drivers/md/dm-writeboost.h
new file mode 100644
index 0000000..3e37b53
--- /dev/null
+++ b/drivers/md/dm-writeboost.h
@@ -0,0 +1,464 @@
+/*
+ * Copyright (C) 2012-2014 Akira Hayakawa <ruby.wktk at gmail.com>
+ *
+ * This file is released under the GPL.
+ */
+
+#ifndef DM_WRITEBOOST_H
+#define DM_WRITEBOOST_H
+
+#define DM_MSG_PREFIX "writeboost"
+
+#include <linux/module.h>
+#include <linux/version.h>
+#include <linux/list.h>
+#include <linux/slab.h>
+#include <linux/vmalloc.h>
+#include <linux/mutex.h>
+#include <linux/kthread.h>
+#include <linux/sched.h>
+#include <linux/timer.h>
+#include <linux/workqueue.h>
+#include <linux/device-mapper.h>
+#include <linux/dm-io.h>
+
+/*----------------------------------------------------------------*/
+
+#define SUB_ID(x, y) ((x) > (y) ? (x) - (y) : 0)
+
+/*----------------------------------------------------------------*/
+
+/*
+ * Nice printk macros
+ *
+ * Production code should not include lineno
+ * but name of the caller seems to be OK.
+ */
+
+/*
+ * Only for debugging.
+ * Don't include this macro in the production code.
+ */
+#define wbdebug(f, args...) \
+ DMINFO("debug@%s() L.%d " f, __func__, __LINE__, ## args)
+
+#define WBERR(f, args...) \
+ DMERR("err@%s() " f, __func__, ## args)
+#define WBWARN(f, args...) \
+ DMWARN("warn@%s() " f, __func__, ## args)
+#define WBINFO(f, args...) \
+ DMINFO("info@%s() " f, __func__, ## args)
+
+/*----------------------------------------------------------------*/
+
+/*
+ * The Detail of the Disk Format (SSD)
+ * -----------------------------------
+ *
+ * ### Overall
+ * Superblock (1MB) + Segment + Segment ...
+ *
+ * ### Superblock
+ * head <---- ----> tail
+ * superblock header (512B) + ... + superblock record (512B)
+ *
+ * ### Segment
+ * segment_header_device (512B) +
+ * metablock_device * nr_caches_inseg +
+ * data[0] (4KB) + data[1] + ... + data[nr_cache_inseg - 1]
+ */
+
+/*----------------------------------------------------------------*/
+
+/*
+ * Superblock Header (Immutable)
+ * -----------------------------
+ * First one sector of the super block region whose value
+ * is unchanged after formatted.
+ */
+#define WB_MAGIC 0x57427374 /* Magic number "WBst" */
+struct superblock_header_device {
+ __le32 magic;
+ __u8 segment_size_order;
+} __packed;
+
+/*
+ * Superblock Record (Mutable)
+ * ---------------------------
+ * Last one sector of the superblock region.
+ * Record the current cache status if required.
+ */
+struct superblock_record_device {
+ __le64 last_migrated_segment_id;
+} __packed;
+
+/*----------------------------------------------------------------*/
+
+/*
+ * The size must be a factor of one sector to avoid starddling
+ * neighboring two sectors.
+ * Facebook's flashcache does the same thing.
+ */
+struct metablock_device {
+ __le64 sector;
+ __u8 dirty_bits;
+ __u8 padding[16 - (8 + 1)]; /* 16B */
+} __packed;
+
+#define WB_CKSUM_SEED (~(u32)0)
+
+struct segment_header_device {
+ /*
+ * We assume 1 sector write is atomic.
+ * This 1 sector region contains important information
+ * such as checksum of the rest of the segment data.
+ * We use 32bit checksum to audit if the segment is
+ * correctly written to the cache device.
+ */
+ /* - FROM ------------------------------------ */
+ __le64 id;
+ /* TODO add timestamp? */
+ __le32 checksum;
+ /*
+ * The number of metablocks in this segment header
+ * to be considered in log replay. The rest are ignored.
+ */
+ __u8 length;
+ __u8 padding[512 - (8 + 4 + 1)]; /* 512B */
+ /* - TO -------------------------------------- */
+ struct metablock_device mbarr[0]; /* 16B * N */
+} __packed;
+
+/*----------------------------------------------------------------*/
+
+struct metablock {
+ sector_t sector; /* The original aligned address */
+
+ u32 idx; /* Index in the metablock array. Const */
+
+ struct hlist_node ht_list; /* Linked to the Hash table */
+
+ u8 dirty_bits; /* 8bit for dirtiness in sector granularity */
+};
+
+#define SZ_MAX (~(size_t)0)
+struct segment_header {
+ u64 id; /* Must be initialized to 0 */
+
+ /*
+ * The number of metablocks in a segment to flush and then migrate.
+ */
+ u8 length;
+
+ u32 start_idx; /* Const */
+ sector_t start_sector; /* Const */
+
+ atomic_t nr_inflight_ios;
+
+ struct metablock mb_array[0];
+};
+
+/*----------------------------------------------------------------*/
+
+enum RAMBUF_TYPE {
+ BUF_NORMAL = 0, /* Volatile DRAM */
+ BUF_NV_BLK, /* Non-volatile with block I/F */
+ BUF_NV_RAM, /* Non-volatile with PRAM I/F */
+};
+
+/*
+ * RAM buffer is a buffer that any dirty data are first written to.
+ * type member in wb_device indicates the buffer type.
+ */
+struct rambuffer {
+ void *data; /* The DRAM buffer. Used as the buffer to submit I/O */
+};
+
+/*
+ * wbflusher's favorite food.
+ * foreground queues this object and wbflusher later pops
+ * one job to submit journal write to the cache device.
+ */
+struct flush_job {
+ struct work_struct work;
+ struct wb_device *wb;
+ struct segment_header *seg;
+ struct rambuffer *rambuf; /* RAM buffer to flush */
+ struct bio_list barrier_ios; /* List of deferred bios */
+};
+
+/*----------------------------------------------------------------*/
+
+enum STATFLAG {
+ STAT_WRITE = 0,
+ STAT_HIT,
+ STAT_ON_BUFFER,
+ STAT_FULLSIZE,
+};
+#define STATLEN (1 << 4)
+
+enum WB_FLAG {
+ /*
+ * This flag is set when either one of the underlying devices
+ * returned EIO and we must immediately block up the whole to
+ * avoid further damage.
+ */
+ WB_DEAD = 0,
+};
+
+/*
+ * The context of the cache driver.
+ */
+struct wb_device {
+ enum RAMBUF_TYPE type;
+
+ struct dm_target *ti;
+
+ struct dm_dev *origin_dev; /* Slow device (HDD) */
+ struct dm_dev *cache_dev; /* Fast device (SSD) */
+
+ mempool_t *buf_1_pool; /* 1 sector buffer pool */
+ mempool_t *buf_8_pool; /* 8 sector buffer pool */
+
+ /*
+ * Mutex is very light-weight.
+ * To mitigate the overhead of the locking we chose to
+ * use mutex.
+ * To optimize the read path, rw_semaphore is an option
+ * but it means to sacrifice write path.
+ */
+ struct mutex io_lock;
+
+ spinlock_t lock;
+
+ u8 segment_size_order; /* Const */
+ u8 nr_caches_inseg; /* Const */
+
+ /*---------------------------------------------*/
+
+ /******************
+ * Current position
+ ******************/
+
+ /*
+ * Current metablock index
+ * which is the last place already written
+ * *not* the position to write hereafter.
+ */
+ u32 cursor;
+ struct segment_header *current_seg;
+ struct rambuffer *current_rambuf;
+
+ /*---------------------------------------------*/
+
+ /**********************
+ * Segment header array
+ **********************/
+
+ u32 nr_segments; /* Const */
+ struct large_array *segment_header_array;
+
+ /*---------------------------------------------*/
+
+ /********************
+ * Chained Hash table
+ ********************/
+
+ u32 nr_caches; /* Const */
+ struct large_array *htable;
+ size_t htsize;
+ struct ht_head *null_head;
+
+ /*---------------------------------------------*/
+
+ /*****************
+ * RAM buffer pool
+ *****************/
+
+ u32 rambuf_pool_amount; /* kB */
+ u32 nr_rambuf_pool; /* Const */
+ struct rambuffer *rambuf_pool;
+ mempool_t *flush_job_pool;
+
+ /*---------------------------------------------*/
+
+ /***********
+ * wbflusher
+ ***********/
+
+ struct workqueue_struct *flusher_wq;
+ wait_queue_head_t flush_wait_queue; /* wait for a segment to be flushed */
+ atomic64_t last_flushed_segment_id;
+
+ /*---------------------------------------------*/
+
+ /*************************
+ * Barrier deadline worker
+ *************************/
+
+ struct work_struct barrier_deadline_work;
+ struct timer_list barrier_deadline_timer;
+ struct bio_list barrier_ios; /* List of barrier requests */
+ unsigned long barrier_deadline_ms; /* tunable */
+
+ /*---------------------------------------------*/
+
+ /****************
+ * Migrate daemon
+ ****************/
+
+ struct task_struct *migrate_daemon;
+ int allow_migrate;
+ int urge_migrate; /* Start migration immediately */
+ int force_drop; /* Don't stop migration */
+ atomic64_t last_migrated_segment_id;
+
+ /*
+ * Data structures used by migrate daemon
+ */
+ wait_queue_head_t migrate_wait_queue; /* wait for a segment to be migrated */
+ wait_queue_head_t wait_drop_caches; /* wait for drop_caches */
+
+ wait_queue_head_t migrate_io_wait_queue; /* wait for migrate ios */
+ atomic_t migrate_io_count;
+ atomic_t migrate_fail_count;
+
+ u32 nr_cur_batched_migration;
+ u32 nr_max_batched_migration; /* tunable */
+
+ u32 num_emigrates; /* Number of emigrates */
+ struct segment_header **emigrates; /* Segments to be migrated */
+ void *migrate_buffer; /* Memorizes the data blocks of the emigrates */
+ u8 *dirtiness_snapshot; /* Memorizes the dirtiness of the metablocks to be migrated */
+
+ /*---------------------------------------------*/
+
+ /*********************
+ * Migration modulator
+ *********************/
+
+ struct task_struct *modulator_daemon;
+ int enable_migration_modulator; /* tunable */
+ u8 migrate_threshold;
+
+ /*---------------------------------------------*/
+
+ /*********************
+ * Superblock recorder
+ *********************/
+
+ struct task_struct *recorder_daemon;
+ unsigned long update_record_interval; /* tunable */
+
+ /*---------------------------------------------*/
+
+ /*************
+ * Sync daemon
+ *************/
+
+ struct task_struct *sync_daemon;
+ unsigned long sync_interval; /* tunable */
+
+ /*---------------------------------------------*/
+
+ /************
+ * Statistics
+ ************/
+
+ atomic64_t nr_dirty_caches;
+ atomic64_t stat[STATLEN];
+ atomic64_t count_non_full_flushed;
+
+ /*---------------------------------------------*/
+
+ unsigned long flags;
+ bool should_emit_tunables; /* should emit tunables in dmsetup table? */
+};
+
+/*----------------------------------------------------------------*/
+
+void acquire_new_seg(struct wb_device *, u64 id);
+void flush_current_buffer(struct wb_device *);
+void inc_nr_dirty_caches(struct wb_device *);
+void cleanup_mb_if_dirty(struct wb_device *, struct segment_header *, struct metablock *);
+u8 read_mb_dirtiness(struct wb_device *, struct segment_header *, struct metablock *);
+void invalidate_previous_cache(struct wb_device *, struct segment_header *,
+ struct metablock *old_mb, bool overwrite_fullsize);
+
+/*----------------------------------------------------------------*/
+
+extern struct workqueue_struct *safe_io_wq;
+extern struct dm_io_client *wb_io_client;
+
+/*
+ * Wrapper of dm_io function.
+ * Set thread to true to run dm_io in other thread to avoid potential deadlock.
+ */
+#define dm_safe_io(io_req, num_regions, regions, err_bits, thread) \
+ dm_safe_io_internal(wb, (io_req), (num_regions), (regions), \
+ (err_bits), (thread), __func__);
+int dm_safe_io_internal(struct wb_device *, struct dm_io_request *,
+ unsigned num_regions, struct dm_io_region *,
+ unsigned long *err_bits, bool thread, const char *caller);
+
+sector_t dm_devsize(struct dm_dev *);
+
+/*----------------------------------------------------------------*/
+
+/*
+ * Device blockup
+ * --------------
+ *
+ * I/O error on either backing device or cache device should block
+ * up the whole system immediately.
+ * After the system is blocked up all the I/Os to underlying
+ * devices are all ignored as if they are switched to /dev/null.
+ */
+
+#define LIVE_DEAD(proc_live, proc_dead) \
+ do { \
+ if (likely(!test_bit(WB_DEAD, &wb->flags))) { \
+ proc_live; \
+ } else { \
+ proc_dead; \
+ } \
+ } while (0)
+
+#define noop_proc do {} while (0)
+#define LIVE(proc) LIVE_DEAD(proc, noop_proc);
+#define DEAD(proc) LIVE_DEAD(noop_proc, proc);
+
+/*
+ * Macro to add context of failure to I/O routine call.
+ * We inherited the idea from Maybe monad of the Haskell language.
+ *
+ * Policies
+ * --------
+ * 1. Only -EIO will block up the system.
+ * 2. -EOPNOTSUPP could be returned if the target device is a virtual
+ * device and we request discard to the device.
+ * 3. -ENOMEM could be returned from blkdev_issue_discard (3.12-rc5)
+ * for example. Waiting for a while can make room for new allocation.
+ * 4. For other unknown error codes we ignore them and ask the users to report.
+ */
+#define IO(proc) \
+ do { \
+ r = 0; \
+ LIVE(r = proc); /* do nothing after blockup */ \
+ if (r == -EOPNOTSUPP) { \
+ r = 0; \
+ } else if (r == -EIO) { \
+ set_bit(WB_DEAD, &wb->flags); \
+ WBERR("device is marked as dead"); \
+ } else if (r == -ENOMEM) { \
+ WBERR("I/O failed by ENOMEM"); \
+ schedule_timeout_interruptible(msecs_to_jiffies(1000));\
+ } else if (r) { \
+ r = 0;\
+ WARN_ONCE(1, "PLEASE REPORT!!! I/O FAILED FOR UNKNOWN REASON err(%d)", r); \
+ } \
+ } while (r)
+
+/*----------------------------------------------------------------*/
+
+#endif
--
1.8.3.4
More information about the dm-devel
mailing list