diff --git a/apps/emqx_durable_storage/BSL.txt b/apps/emqx_durable_storage/BSL.txt deleted file mode 100644 index f0cd31c6f..000000000 --- a/apps/emqx_durable_storage/BSL.txt +++ /dev/null @@ -1,94 +0,0 @@ -Business Source License 1.1 - -Licensor: Hangzhou EMQ Technologies Co., Ltd. -Licensed Work: EMQX Enterprise Edition - The Licensed Work is (c) 2023 - Hangzhou EMQ Technologies Co., Ltd. -Additional Use Grant: Students and educators are granted right to copy, - modify, and create derivative work for research - or education. -Change Date: 2028-01-26 -Change License: Apache License, Version 2.0 - -For information about alternative licensing arrangements for the Software, -please contact Licensor: https://www.emqx.com/en/contact - -Notice - -The Business Source License (this document, or the “License”) is not an Open -Source license. However, the Licensed Work will eventually be made available -under an Open Source License, as stated in this License. - -License text copyright (c) 2017 MariaDB Corporation Ab, All Rights Reserved. -“Business Source License” is a trademark of MariaDB Corporation Ab. - ------------------------------------------------------------------------------ - -Business Source License 1.1 - -Terms - -The Licensor hereby grants you the right to copy, modify, create derivative -works, redistribute, and make non-production use of the Licensed Work. The -Licensor may make an Additional Use Grant, above, permitting limited -production use. - -Effective on the Change Date, or the fourth anniversary of the first publicly -available distribution of a specific version of the Licensed Work under this -License, whichever comes first, the Licensor hereby grants you rights under -the terms of the Change License, and the rights granted in the paragraph -above terminate. - -If your use of the Licensed Work does not comply with the requirements -currently in effect as described in this License, you must purchase a -commercial license from the Licensor, its affiliated entities, or authorized -resellers, or you must refrain from using the Licensed Work. - -All copies of the original and modified Licensed Work, and derivative works -of the Licensed Work, are subject to this License. This License applies -separately for each version of the Licensed Work and the Change Date may vary -for each version of the Licensed Work released by Licensor. - -You must conspicuously display this License on each original or modified copy -of the Licensed Work. If you receive the Licensed Work in original or -modified form from a third party, the terms and conditions set forth in this -License apply to your use of that work. - -Any use of the Licensed Work in violation of this License will automatically -terminate your rights under this License for the current and all other -versions of the Licensed Work. - -This License does not grant you any right in any trademark or logo of -Licensor or its affiliates (provided that you may use a trademark or logo of -Licensor as expressly required by this License). - -TO THE EXTENT PERMITTED BY APPLICABLE LAW, THE LICENSED WORK IS PROVIDED ON -AN “AS IS” BASIS. LICENSOR HEREBY DISCLAIMS ALL WARRANTIES AND CONDITIONS, -EXPRESS OR IMPLIED, INCLUDING (WITHOUT LIMITATION) WARRANTIES OF -MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NON-INFRINGEMENT, AND -TITLE. - -MariaDB hereby grants you permission to use this License’s text to license -your works, and to refer to it using the trademark “Business Source License”, -as long as you comply with the Covenants of Licensor below. - -Covenants of Licensor - -In consideration of the right to use this License’s text and the “Business -Source License” name and trademark, Licensor covenants to MariaDB, and to all -other recipients of the licensed work to be provided by Licensor: - -1. To specify as the Change License the GPL Version 2.0 or any later version, - or a license that is compatible with GPL Version 2.0 or a later version, - where “compatible” means that software provided under the Change License can - be included in a program with software provided under GPL Version 2.0 or a - later version. Licensor may specify additional Change Licenses without - limitation. - -2. To either: (a) specify an additional grant of rights to use that does not - impose any additional restriction on the right granted in this License, as - the Additional Use Grant; or (b) insert the text “None”. - -3. To specify a Change Date. - -4. Not to modify this License in any other way. diff --git a/apps/emqx_durable_storage/IMPLEMENTATION.md b/apps/emqx_durable_storage/IMPLEMENTATION.md deleted file mode 100644 index 33f02dfc4..000000000 --- a/apps/emqx_durable_storage/IMPLEMENTATION.md +++ /dev/null @@ -1,119 +0,0 @@ -# General concepts - -In the logic layer we don't speak about replication. -This is because we could use an external DB with its own replication logic. - -On the other hand, we introduce notion of shard right here at the logic layer. -This is because shared subscription logic needs to be aware of it to some extend, as it has to split work between subscribers somehow. - - -# Modus operandi - -1. Create a draft implementation of a milestone -2. Test performance -3. Test consistency -4. Goto 1 - -# Tables - -## Message storage - -Data is written every time a message matching certain pattern is published. -This pattern is not part of the logic layer spec. - -Write throughput: very high - -Data size: very high - -Write pattern: append only - -Read pattern: pseudoserial - -Number of records: O(total write throughput * retention time) - - -# Push vs. Pull model - -In push model we have replay agents iterating over the dataset in the shards. - -In pull model the client processes work with iterators directly and fetch data from the remote message storage instances via remote procedure calls. - -## Push pros: -- Lower latency: message can be dispatched to the client as soon as it's persisted -- Less worry about buffering - -## Push cons: -- Needs pushback logic -- It's not entirely justified when working with external DB that may not provide streaming API - -## Pull pros: -- No need for pushback: client advances iterators at its own tempo - -## Pull cons: -- 2 messages need to be sent over network for each batch being replayed. -- RPC is generally an antipattern - -# Invariants - -- All messages written to the shard always have larger sequence number than all the iterators for the shard (to avoid missing messages during replay) - - -# Parallel tracks - -## 0. Configuration - -This includes HOCON schema and an interface module that is used by the rest of the code (`emqx_ds_conf.erl`). - -At the early stage we need at least to implement a feature flag that can be used by developers. - -We should have safety measures to prevent a client from breaking down the broker by connecting with clean session = false and subscribing to `#`. - -## 1. Fully implement all emqx_durable_storage APIs - -### Message API - -### Session API - -### Iterator API - -## 2. Implement a filter for messages that has to be persisted - -We don't want to persist ALL messages. -Only the messages that can be replayed by some client. - -1. Persistent sessions should signal to the emqx_broker what topic filters should be persisted. - Scenario: - - Client connects with clean session = false. - - Subscribes to the topic filter a/b/# - - Now we need to signal to the rest of the broker that messages matching `a/b/#` must be persisted. - -2. Replay feature (separate, optional, under BSL license): in the configuration file we have list of topic filters that specify what topics can be replayed. (Lower prio) - - Customers can do this: `#` - - Include minimum QoS of messages in the config - -## 3. Replace current emqx_persistent_session with the emqx_durable_storage - -## 4. Garbage collection - -## 5. Tooling for performance and consistency testing - -At the first stage we can just use emqttb: - -https://github.com/emqx/emqttb/blob/master/src/scenarios/emqttb_scenario_persistent_session.erl - - -Consistency verification at the early stages can just use this test suite: - -`apps/emqx/test/emqx_persistent_session_SUITE.erl` - -## 6. Update rocksdb version in EMQX (low prio) - -## 9999. Alternative schema for rocksdb message table - -https://github.com/emqx/eip/blob/main/active/0023-rocksdb-message-persistence.md#potential-for-future-optimizations-keyspace-based-on-the-learned-topic-patterns - -Problem with the current bitmask-based schema: - -- Good: `a/b/c/#` - -- Bad: `+/a/b/c/d` diff --git a/apps/emqx_durable_storage/src/emqx_ds_app.erl b/apps/emqx_durable_storage/src/emqx_ds_app.erl index dcf353a99..552d6e044 100644 --- a/apps/emqx_durable_storage/src/emqx_ds_app.erl +++ b/apps/emqx_durable_storage/src/emqx_ds_app.erl @@ -1,5 +1,17 @@ %%-------------------------------------------------------------------- %% Copyright (c) 2020-2024 EMQ Technologies Co., Ltd. All Rights Reserved. +%% +%% Licensed under the Apache License, Version 2.0 (the "License"); +%% you may not use this file except in compliance with the License. +%% You may obtain a copy of the License at +%% +%% http://www.apache.org/licenses/LICENSE-2.0 +%% +%% Unless required by applicable law or agreed to in writing, software +%% distributed under the License is distributed on an "AS IS" BASIS, +%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +%% See the License for the specific language governing permissions and +%% limitations under the License. %%-------------------------------------------------------------------- -module(emqx_ds_app). diff --git a/apps/emqx_durable_storage/src/emqx_ds_bitmask_keymapper.erl b/apps/emqx_durable_storage/src/emqx_ds_bitmask_keymapper.erl index a3b65c7e6..11dcba3ea 100644 --- a/apps/emqx_durable_storage/src/emqx_ds_bitmask_keymapper.erl +++ b/apps/emqx_durable_storage/src/emqx_ds_bitmask_keymapper.erl @@ -1,5 +1,17 @@ %%-------------------------------------------------------------------- -%% Copyright (c) 2022-2023 EMQ Technologies Co., Ltd. All Rights Reserved. +%% Copyright (c) 2022-2024 EMQ Technologies Co., Ltd. All Rights Reserved. +%% +%% Licensed under the Apache License, Version 2.0 (the "License"); +%% you may not use this file except in compliance with the License. +%% You may obtain a copy of the License at +%% +%% http://www.apache.org/licenses/LICENSE-2.0 +%% +%% Unless required by applicable law or agreed to in writing, software +%% distributed under the License is distributed on an "AS IS" BASIS, +%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +%% See the License for the specific language governing permissions and +%% limitations under the License. %%-------------------------------------------------------------------- -module(emqx_ds_bitmask_keymapper). diff --git a/apps/emqx_durable_storage/src/emqx_ds_builtin_db_sup.erl b/apps/emqx_durable_storage/src/emqx_ds_builtin_db_sup.erl index 9df07eb18..68aa0ee90 100644 --- a/apps/emqx_durable_storage/src/emqx_ds_builtin_db_sup.erl +++ b/apps/emqx_durable_storage/src/emqx_ds_builtin_db_sup.erl @@ -1,5 +1,17 @@ %%-------------------------------------------------------------------- %% Copyright (c) 2024 EMQ Technologies Co., Ltd. All Rights Reserved. +%% +%% Licensed under the Apache License, Version 2.0 (the "License"); +%% you may not use this file except in compliance with the License. +%% You may obtain a copy of the License at +%% +%% http://www.apache.org/licenses/LICENSE-2.0 +%% +%% Unless required by applicable law or agreed to in writing, software +%% distributed under the License is distributed on an "AS IS" BASIS, +%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +%% See the License for the specific language governing permissions and +%% limitations under the License. %%-------------------------------------------------------------------- %% @doc Supervisor that contains all the processes that belong to a diff --git a/apps/emqx_durable_storage/src/emqx_ds_storage_layer_sup.erl b/apps/emqx_durable_storage/src/emqx_ds_storage_layer_sup.erl index 424b35133..136669ed2 100644 --- a/apps/emqx_durable_storage/src/emqx_ds_storage_layer_sup.erl +++ b/apps/emqx_durable_storage/src/emqx_ds_storage_layer_sup.erl @@ -1,5 +1,17 @@ %%-------------------------------------------------------------------- %% Copyright (c) 2022-2024 EMQ Technologies Co., Ltd. All Rights Reserved. +%% +%% Licensed under the Apache License, Version 2.0 (the "License"); +%% you may not use this file except in compliance with the License. +%% You may obtain a copy of the License at +%% +%% http://www.apache.org/licenses/LICENSE-2.0 +%% +%% Unless required by applicable law or agreed to in writing, software +%% distributed under the License is distributed on an "AS IS" BASIS, +%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +%% See the License for the specific language governing permissions and +%% limitations under the License. %%-------------------------------------------------------------------- -module(emqx_ds_storage_layer_sup). diff --git a/apps/emqx_durable_storage/src/emqx_ds_sup.erl b/apps/emqx_durable_storage/src/emqx_ds_sup.erl index e863e74ce..ed2c83c62 100644 --- a/apps/emqx_durable_storage/src/emqx_ds_sup.erl +++ b/apps/emqx_durable_storage/src/emqx_ds_sup.erl @@ -1,5 +1,17 @@ %%-------------------------------------------------------------------- %% Copyright (c) 2022-2024 EMQ Technologies Co., Ltd. All Rights Reserved. +%% +%% Licensed under the Apache License, Version 2.0 (the "License"); +%% you may not use this file except in compliance with the License. +%% You may obtain a copy of the License at +%% +%% http://www.apache.org/licenses/LICENSE-2.0 +%% +%% Unless required by applicable law or agreed to in writing, software +%% distributed under the License is distributed on an "AS IS" BASIS, +%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +%% See the License for the specific language governing permissions and +%% limitations under the License. %%-------------------------------------------------------------------- -module(emqx_ds_sup). diff --git a/apps/emqx_durable_storage/test/emqx_ds_storage_bitfield_lts_SUITE.erl b/apps/emqx_durable_storage/test/emqx_ds_storage_bitfield_lts_SUITE.erl index 349eede86..72bb70949 100644 --- a/apps/emqx_durable_storage/test/emqx_ds_storage_bitfield_lts_SUITE.erl +++ b/apps/emqx_durable_storage/test/emqx_ds_storage_bitfield_lts_SUITE.erl @@ -1,5 +1,17 @@ %%-------------------------------------------------------------------- %% Copyright (c) 2022-2024 EMQ Technologies Co., Ltd. All Rights Reserved. +%% +%% Licensed under the Apache License, Version 2.0 (the "License"); +%% you may not use this file except in compliance with the License. +%% You may obtain a copy of the License at +%% +%% http://www.apache.org/licenses/LICENSE-2.0 +%% +%% Unless required by applicable law or agreed to in writing, software +%% distributed under the License is distributed on an "AS IS" BASIS, +%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +%% See the License for the specific language governing permissions and +%% limitations under the License. %%-------------------------------------------------------------------- -module(emqx_ds_storage_bitfield_lts_SUITE). diff --git a/apps/emqx_durable_storage/test/props/emqx_ds_message_storage_bitmask_shim.erl b/apps/emqx_durable_storage/test/props/emqx_ds_message_storage_bitmask_shim.erl index ddc199a44..97b90fcb7 100644 --- a/apps/emqx_durable_storage/test/props/emqx_ds_message_storage_bitmask_shim.erl +++ b/apps/emqx_durable_storage/test/props/emqx_ds_message_storage_bitmask_shim.erl @@ -1,5 +1,17 @@ %%-------------------------------------------------------------------- -%% Copyright (c) 2020-2023 EMQ Technologies Co., Ltd. All Rights Reserved. +%% Copyright (c) 2020-2024 EMQ Technologies Co., Ltd. All Rights Reserved. +%% +%% Licensed under the Apache License, Version 2.0 (the "License"); +%% you may not use this file except in compliance with the License. +%% You may obtain a copy of the License at +%% +%% http://www.apache.org/licenses/LICENSE-2.0 +%% +%% Unless required by applicable law or agreed to in writing, software +%% distributed under the License is distributed on an "AS IS" BASIS, +%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +%% See the License for the specific language governing permissions and +%% limitations under the License. %%-------------------------------------------------------------------- -module(emqx_ds_message_storage_bitmask_shim). diff --git a/apps/emqx_durable_storage/test/props/payload_gen.erl b/apps/emqx_durable_storage/test/props/payload_gen.erl index b969c0043..d4d19b403 100644 --- a/apps/emqx_durable_storage/test/props/payload_gen.erl +++ b/apps/emqx_durable_storage/test/props/payload_gen.erl @@ -1,3 +1,19 @@ +%%-------------------------------------------------------------------- +%% Copyright (c) 2023-2024 EMQ Technologies Co., Ltd. All Rights Reserved. +%% +%% Licensed under the Apache License, Version 2.0 (the "License"); +%% you may not use this file except in compliance with the License. +%% You may obtain a copy of the License at +%% +%% http://www.apache.org/licenses/LICENSE-2.0 +%% +%% Unless required by applicable law or agreed to in writing, software +%% distributed under the License is distributed on an "AS IS" BASIS, +%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +%% See the License for the specific language governing permissions and +%% limitations under the License. +%%-------------------------------------------------------------------- + %% @doc This module provides lazy, composable producer streams that %% can be considered counterparts to Archiver's consumer pipes and %% therefore can facilitate testing