qurel / qurel.github.io Goto Github PK

https://qurel.github.io/python-hook/

python代码中捕捉requests网络请求

import sys
import requests

def trace_calls(frame, event, arg):
if event != 'call':
return
co = frame.f_code
arr = ['url','params','data','body']
func_name = co.co_name
if func_name == 'request':
for key, value in frame.f_locals.items():
if key not in arr: continue
if type(value) is bytes: value = value.decode('unicode-escape')
print(f

https://qurel.github.io/backup-types/

VM backup definition

Virtual machine backup (VM backup) is the process of backing up the virtual machines (VMs) running in an enterprise environment.

VMs usually run as guests on hypervisors that emulate a computer system, and allow multiple VMs to share a physical host hardware system.

As enterprises increasingly rely on virtualization, VMs have become a vital part of enterprise IT environments. Business applications databases, and even containerized workloads run on VMs and generate vast amounts of data that need to be protected using a robust data protection solution. The applications that enable companies to backup and restore all the files that comprise entire virtual machines are called VM backup software.

What is VM backup?

虚拟机备份是虚拟机的数据保护解决方案，它执行与物理服务器使用的传统备份解决方案类似的功能。

Virtual machine backup applications can perform a full backup of all files in a VM, an incremental backup, or differential backup.

vm备份软件需要频繁和定期运行，以保护vm文件、配置和不断变化的数据。

现代的vm备份软件提供更强大的功能，允许进行更快且对虚拟机影响更小的备份。

VM backup concepts

Differential backup – 备份从上次全量备份以来变化的文件。
File-level backup – Backup that is defined at the level of files and folders.
Full backup – A backup of all files.
Full vm backup – 构成整个虚拟机的所有文件的备份，包括磁盘映像、配置文件等。
Image-level backup – or volume level 备份整个存储卷
Incremental backup – Backup of only the files that have changed since the last backup, either full or incremental.
Quiescing – A process to bring the data of the virtual machine to a state suitable for backups including, flushing of buffers from the operating systems memory cache to disk or other application specific tasks.

运行在虚拟机内部的备份客户端

虚拟机如同物理机一样，可以使用运行在系统内部的备份软件进行备份。采用这种办法，备份客户端可以在虚拟机进行备份的时候静默其系统。这种方法典型的应用是，对虚拟机内部磁盘存储数据进行文件层次的备份。

This method has advantages and disadvantages. Advantages can be there are no procedural changes or skills required since the backup agent is similar process to that of a physical server backup. This method can help with consistency of application data for some business critical workloads like databases. A disadvantage could be higher usage of the hosts resources as the backup operation is being carried out. However, computer resources today are more capable and the backup software is less resource intensive.

运行在宿主机的备份客户端

因为虚拟机事实上就是由若干个配置、镜像文件等组成，所以运行在宿主机的vm管理程序可以对其进行备份。 This allows for the files that make up the virtual machine to be backed up as files, but with a few differences. Since these files are being written to while the virtual machine is running, the VM must be power off.

If your VM backups include transactional apps such as database and email servers, quiescing them to make sure that they are in a proper state for backup is referred to as application-consistent as an example. Before a backup starts, the application is paused to ensure that any outstanding transactions and writes are written to disk. This step ensures that the server is aware of the operation and that no data will be lost if VM recovery is needed. Quiescing only works with applications that support the pausing and writing of pending data whenever necessary.（对应用的）静默，只能工作在那些支持挂起和恢复数据的应用之上（MySQL:FTWRL）.

vm backup的应用场景

业务的连续性。借助 vm 备份软件，企业可以保护其所有数据和配置，以便在发生不可预测的中断后恢复业务。
数据保护。如果没有适当的保护，数据容易受到潜在的丢失和损坏。vm backup可保留数据的完整性并提供丢失数据的可用副本。
灾难恢复。众所周知，所有企业和 IT 环境都将面临可能导致数据丢失、损坏或中断 IT 运营的意外事件。使用vm backup软件可降低降低中断的风险。

静默备份

一致性备份

  一致性
  细节
  恢复
   




  崩溃一致性 Crash-consistent
  表示备份的数据类似系统异常宕机时的状态（应用对文件的写还没有停止，应用的业务操作未全部flush，如数据库事务）
  vm启动，然后进行磁盘检测然后恢复损坏错误。应用程序实现自己的数据验证。例如，数据库应用程序可以使用其事务日志进行验证。如果事务日志中的条目不在数据库中，数据库将回滚事务，直到数据一致。
   


  文件系统一致性 File-system consistent
  文件系统一致性备份情况下，文件系统缓存中的数据应该刷入磁盘中，文件备份处于同一时刻。
  应用程序需要实现自己的“修复”机制，以确保恢复的数据是一致的。
   


  应用一致性 Application-consistent
  表示备份的数据是应用可用的数据；（应用写完成，如事务完成）
  没有数据损坏或丢失。应用程序以一致的状态启动。
   


   
   
   
   

图示

以运行MySQL的Linux系统为例，进行文件系统一致性备份的大致流程为：

https://qurel.github.io/qemu-coroutine/

非局部转移

解析

qemu-coroutine的ucontext模式是基于setjmp和longjmp实现的，可以完成函数外部的跳转。

setjmp示例：

#include <stdio.h>
#include <setjmp.h>

jmp_buf env;

int func1()
{
printf(

https://qurel.github.io/hbitmap/?

常用的位运算

(x + ( 1 « granularity - 1) ) » granularity = x / 2^granularity ,且若后granularity位某位不为1，都进1
x & (x - 1) : x最后一位1置0，如x = 0b01100，则x & (x - 1) = 0b01000

数据结构

struct HBitmap {
/* Number of total bits in the bottom level. /
/
size = (size + (1ULL << hb->granularity) - 1) >> hb->granularity;
如虚拟镜像大小为1G，size = (102410241024 + (1<<16) -1 ) >>16 即size=16385
*/
uint64_t size;

/* Number of set bits in the bottom level.  */
uint64_t count;

/* A scaling factor.  Given a granularity of G, each bit in the bitmap will
 * will actually represent a group of 2^G elements.  Each operation on a
 * range of bits first rounds the bits to determine which group they land
 * in, and then affect the entire page; iteration will only visit the first
 * bit of each group.  Here is an example of operations in a size-16,
 * granularity-1 HBitmap:
 *
 *    initial state            00000000
 *    set(start=0, count=9)    11111000 (iter: 0, 2, 4, 6, 8)
 *    reset(start=1, count=3)  00111000 (iter: 4, 6, 8)
 *    set(start=9, count=2)    00111100 (iter: 4, 6, 8, 10)
 *    reset(start=5, count=5)  00000000
 *
 * From an implementation point of view, when setting or resetting bits,
 * the bitmap will scale bit numbers right by this amount of bits.  When
 * iterating, the bitmap will scale bit numbers left by this amount of
 * bits.
 */
// 粒度 默认16，则bitmap中的每位表示2^16个元素
int granularity;

/* A number of progressively less coarse bitmaps (i.e. level 0 is the
 * coarsest).  Each bit in level N represents a word in level N+1 that
 * has a set bit, except the last level where each bit represents the
 * actual bitmap.
 *
 * Note that all bitmaps have the same number of levels.  Even a 1-bit
 * bitmap will still allocate HBITMAP_LEVELS arrays.
 */
// 每一层的位图，在x64环境下 HBITMAP_LEVELS = 7
unsigned long *levels[HBITMAP_LEVELS];

/* The length of each levels[] array. */
uint64_t sizes[HBITMAP_LEVELS]; //每一曾位图的长度

};

流程

初始化

sizes[HBITMAP_LEVELS]的分配，可见是长度是逐层 除6：

for (i = HBITMAP_LEVELS; i-- > 0; ) {
   //  BITS_PER_LEVEL=6 ,BITS_PER_LONG  = 64 = 2^6
    size = MAX((size + BITS_PER_LONG - 1) >> BITS_PER_LEVEL, 1);
    hb->sizes[i] = size;
    hb->levels[i] = g_new0(unsigned long, size);
}

HBitmap层次结构图，总共7层，下层记录上层的修改bit位，当上层某bit位置为1时，下层会相应的位记录：

hbitmap_set

void hbitmap_set(HBitmap *hb, uint64_t start, uint64_t count)
{
// 终止位置
uint64_t last = start + count - 1;

trace_hbitmap_set(hb, start, count,
                  start >> hb->granularity, last >> hb->granularity);

// 除以粒度
start >>= hb->granularity;
last >>= hb->granularity;
count = last - start + 1;

hb->count += count - hb_count_between(hb, start, last);
// 从第六层开始操作，递归函数
hb_set_between(hb, HBITMAP_LEVELS - 1, start, last);

}

static void hb_set_between(HBitmap *hb, int level, uint64_t start, uint64_t last)
{
// BITS_PER_LEVEL=6,即pos = start/64，因为hb->levels每个元素位uint64_t即64位，pos表示要置的起始位位于第几个word中
size_t pos = start >> BITS_PER_LEVEL;
size_t lastpos = last >> BITS_PER_LEVEL;
bool changed = false;
size_t i;

i = pos;
if (i < lastpos) {
    uint64_t next = (start | (BITS_PER_LONG - 1)) + 1;
    changed |= hb_set_elem(&hb->levels[level][i], start, next - 1);
    for (;;) {
        start = next;
        next += BITS_PER_LONG;
        if (++i == lastpos) {
            break;
        }
        changed |= (hb->levels[level][i] == 0);
        hb->levels[level][i] = ~0UL;
    }
}
// 计算需要置的位位于word中的哪一位
changed |= hb_set_elem(&hb->levels[level][i], start, last);

/* If there was any change in this layer, we may have to update
 * the one above.
 */
if (level > 0 && changed) {
    // 递归计算上一层
    hb_set_between(hb, level - 1, pos, lastpos);
}

}

/* Setting starts at the last layer and propagates up if an element

• changes from zero to non-zero.
  */
  static inline bool hb_set_elem(unsigned long *elem, uint64_t start, uint64_t last)
  {
  unsigned long mask;
  bool changed;

  assert((last >> BITS_PER_LEVEL) == (start >> BITS_PER_LEVEL));
  assert(start <= last);
  // BITS_PER_LONG = 64
  // mask = ( 0b10 << (last % 2^6) ) - ( 0b01 << (start % 2^6) )
  // 例如start = 34976，last = 34979
  // 则 mask = 2 << (34979 % 64) - 1 << (34976 % 64)
  // = 2 << 35 - 1 << 32
  // = 0xf00000000
  // 在word中的第32，33，34，35位分别写入1
  mask = 2UL << (last & (BITS_PER_LONG - 1));
  mask -= 1UL << (start & (BITS_PER_LONG - 1));
  changed = (*elem == 0);
  // 将mask相应的1位置入hb->levels[level][i]
  *elem |= mask;
  return changed;
  }

hbitmap_get

bool hbitmap_get(const HBitmap hb, uint64_t item)
{
/ Compute position and bit in the last layer. */
uint64_t pos = item >> hb->granularity;
unsigned long bit = 1UL << (pos & (BITS_PER_LONG - 1)); // bit = pos % 64,计算初在word中的位置
// BITS_PER_LEVEL =6
// 按位与计算出相应word中的相应位是否位1
return (hb->levels[HBITMAP_LEVELS - 1][pos >> BITS_PER_LEVEL] & bit) != 0;
}

迭代器

数据结构

struct HBitmapIter {
const HBitmap *hb;

/* Copied from hb for access in the inline functions (hb is opaque).  */
int granularity;

/* Entry offset into the last-level array of longs.  */
size_t pos;

/* The currently-active path in the tree.  Each item of cur[i] stores
 * the bits (i.e. the subtrees) yet to be processed under that node.
 */
unsigned long cur[HBITMAP_LEVELS];

};

初始化

void hbitmap_iter_init(HBitmapIter *hbi, const HBitmap *hb, uint64_t first)
{
unsigned i, bit;
uint64_t pos;

hbi->hb = hb;
pos = first >> hb->granularity;
assert(pos < hb->size);
hbi->pos = pos >> BITS_PER_LEVEL; // pos表示最底层word的顺序
hbi->granularity = hb->granularity;

for (i = HBITMAP_LEVELS; i-- > 0; ) { //最底层开始初始化
    bit = pos & (BITS_PER_LONG - 1);// bit = pos%64,位于word中第几位
    pos >>= BITS_PER_LEVEL; //第几个word

    /* Drop bits representing items before first.  */
    //  ~((1UL << bit) - 1) 高于或等于bit的所有位全部置1
    hbi->cur[i] = hb->levels[i][pos] & ~((1UL << bit) - 1);

    /* We have already added level i+1, so the lowest set bit has
     * been processed.  Clear it.
     */
    if (i != HBITMAP_LEVELS - 1) {
        // 最底层正在被迭代，因此其上层的对应位全部置0
        // 这样当在hbitmap_iter_next函数中，当前word为0时，可以直接跳到上层寻找下一个word
        hbi->cur[i] &= ~(1UL << bit);
    }
}

}

迭代

static inline int64_t hbitmap_iter_next(HBitmapIter *hbi)
{
unsigned long cur = hbi->cur[HBITMAP_LEVELS - 1];
int64_t item;

if (cur == 0) {
    // 如果当前word的位全被迭代完，迭代下一个word
    cur = hbitmap_iter_skip_words(hbi);
    if (cur == 0) {
        return -1;
    }
}

/* The next call will resume work from the next bit.  */
hbi->cur[HBITMAP_LEVELS - 1] = cur & (cur - 1); //去掉最尾的一个0，表示该位已经被迭代过
//ctz64 - count trailing zeros in a 64-bit value. ctzl表示末尾几个零，表示word中保存的具体值
item = ((uint64_t)hbi->pos << BITS_PER_LEVEL) + ctzl(cur); //pos*64 + word，获得位图中的位置
// 加上cur尾部0的数可以获得在word中的第几位
return item << hbi->granularity;

}

unsigned long hbitmap_iter_skip_words(HBitmapIter *hbi)
{
size_t pos = hbi->pos;
const HBitmap *hb = hbi->hb;
unsigned i = HBITMAP_LEVELS - 1;

unsigned long cur;
do { //迭代寻找每一层中不为零的，如果最顶层也不为零，则说明bitmap全部被迭代完，没有剩余元素
    cur = hbi->cur[--i];
    pos >>= BITS_PER_LEVEL;
} while (cur == 0);

/* Check for end of iteration.  We always use fewer than BITS_PER_LONG
 * bits in the level 0 bitmap; thus we can repurpose the most significant
 * bit as a sentinel.  The sentinel is set in hbitmap_alloc and ensures
 * that the above loop ends even without an explicit check on i.
 */

if (i == 0 && cur == (1UL << (BITS_PER_LONG - 1))) { //迭代到最顶层，并且最顶层没有置位元素
    return 0;
}
// 找到置位那层，开始寻找下一个迭代位置
for (; i < HBITMAP_LEVELS - 1; i++) {
    /* Shift back pos to the left, matching the right shifts above.
     * The index of this word's least significant set bit provides
     * the low-order bits.
     */
    assert(cur);
    // 逐层恢复pos
    pos = (pos << BITS_PER_LEVEL) + ctzl(cur);
    // cur前进一位
    hbi->cur[i] = cur & (cur - 1);

    /* Set up next level for iteration.  */
    //  cur为当前层的word
    cur = hb->levels[i + 1][pos];
}

hbi->pos = pos;
trace_hbitmap_iter_skip_words(hbi->hb, hbi, pos, cur);

assert(cur);
return cur;

}