- Source: Re-Pair
Re-Pair (short for recursive pairing) is a grammar-based compression algorithm that, given an input text, builds a straight-line program, i.e. a context-free grammar generating a single string: the input text. In order to perform the compression in linear time, it consumes the amount of memory that is approximately five times the size of its input.
The grammar is built by recursively replacing the most frequent pair of characters occurring in the text.
Once there is no pair of characters occurring twice, the resulting string is used as the axiom of the grammar.
Therefore, the output grammar is such that all rules but the axiom have two symbols on the right-hand side.
How it works
Re-Pair was first introduced by NJ. Larsson and A. Moffat in 1999.
In their paper the algorithm is presented together with a detailed description of the data structures required to implement it with linear time and space complexity. The experiments showed that Re-Pair achieves high compression ratios and offers good performance for decompression. However, the major drawback of the algorithm is its memory consumption, which is approximately 5 times the size of the input. Such memory usage is required in order to perform the compression in linear time but makes the algorithm impractical for compressing large files.
The image on the right shows how the algorithm works compresses the string
w
=
x
a
b
c
a
b
c
y
123123
z
a
b
c
{\displaystyle w=xabcabcy123123zabc}
.
During the first iteration, the pair
a
b
{\displaystyle ab}
, which occurs three times in
w
{\displaystyle w}
, is replaced by a new symbol
R
1
{\displaystyle R_{1}}
.
On the second iteration, the most frequent pair in the string
w
=
x
R
1
c
R
1
c
y
123123
z
R
1
c
{\displaystyle w=xR_{1}cR_{1}cy123123zR_{1}c}
, which is
R
1
c
{\displaystyle R_{1}c}
, is replaced by a new symbol
R
2
{\displaystyle R_{2}}
.
Thus, at the end of the second iteration, the remaining string is
w
=
x
R
2
R
2
y
123123
z
R
2
{\displaystyle w=xR_{2}R_{2}y123123zR_{2}}
.
In the next two iterations, the pairs
12
{\displaystyle 12}
and
R
3
3
{\displaystyle R_{3}3}
are replaced by symbols
R
3
{\displaystyle R_{3}}
and
R
4
{\displaystyle R_{4}}
respectively.
Finally, the string
w
=
x
R
2
R
2
y
R
4
R
4
z
R
2
{\displaystyle w=xR_{2}R_{2}yR_{4}R_{4}zR_{2}}
contains no repeated pair and therefore it is used as the axiom of the output grammar.
Data structures
In order to achieve linear time complexity, Re-Pair requires the following data structures
A sequence representing the input string. Position
i
{\displaystyle i}
of the sequence contains the i-th symbol of the input string plus two references to other positions in the sequence. These references point to the next/previous positions, say
k
{\displaystyle k}
and
m
{\displaystyle m}
, such that the same substring begins at
w
[
i
]
{\displaystyle w[i]}
,
w
[
k
]
{\displaystyle w[k]}
and
w
[
m
]
{\displaystyle w[m]}
and all three occurrences are captured by the same reference (i.e. there is a variable in the grammar generating the string).
A priority queue. Each element of the queue is a pair of symbols (terminals or previously defined pairs) that occur consecutively in the sequence. The priority of a pair is given by the number of occurrences of the pair in the remaining sequence. Each time a new pair is created, the priority queue is updated.
A hash table to keep track of already defined pairs. This table is updated each time a new pair is created or removed.
Since the hash table and the priority queue refer to the same elements (pairs), they can be implemented by a common data structure called PAIR with pointers for the hash table (h_next) and the priority queue (p_next and p_prev). Furthermore, each PAIR points to the beginning of the first (f_pos) and the last (b_pos) occurrences of the string represented by the PAIR in the sequence. The following picture shows an overview of this data structure.
The following two pictures show an example of how these data structures look after the initialization and after applying one step of the pairing process (pointers to NULL are not displayed):
= Encoding the grammar
=Once the grammar has been built for a given input string, in order to achieve effective compression, this grammar has to be encoded efficiently.
One of the simplest methods for encoding the grammar is the implicit encoding, which consists on invoking function encodeCFG(X), described below, sequentially on all the axiom's symbols.
Intuitively, rules are encoded as they are visited in a depth-first traversal of the grammar. The first time a rule is visited, its right hand side is encoded recursively and a new code is assigned to the rule. From that point, whenever the rule is reached, the assigned value is written.
Another possibility is to separate the rules of the grammar into generations such that a rule
X
→
Y
Z
{\displaystyle X\to YZ}
belongs to generation
i
{\displaystyle i}
iff at least one of
Y
{\displaystyle Y}
or
Z
{\displaystyle Z}
belongs to generation
i
−
1
{\displaystyle i-1}
and the other belongs to generation
j
{\displaystyle j}
with
j
≤
i
−
1
{\displaystyle j\leq i-1}
. Then these generations are encoded subsequently starting from generation
0
{\displaystyle 0}
. This was the method proposed originally when Re-Pair was first introduced. However, most implementations of Re-Pair use the implicit encoding method due to its simplicity and good performance. Furthermore, it allows on-the-fly decompression.
Versions
There exists a number of different implementations of Re-Pair. Each of these versions aims at improving one specific aspect of the algorithm, such as reducing the runtime, reducing the space consumption or increasing the compression ratio.
See also
Byte pair encoding
Sequitur algorithm
References
Kata Kunci Pencarian:
- Tabel periodik
- Maé-Bérénice Méité
- Texas hold 'em
- Junior MasterChef Indonesia
- The Silent Service
- Transmisi (telekomunikasi)
- Mesin pencari
- Evolusi manusia
- Code Geass: Fukkatsu no Lelouch
- Hewan
- Re-Pair
- Byte pair encoding
- Au pair
- Dirty Pair
- Republican Political Alliance for Integrity and Reform
- Metaphor: ReFantazio
- A Nice Pair
- Beauty and the Geek Australia
- Beauty and the Geek (British TV series)
- Greylag goose
The Bourne Supremacy (2004)
Action, Bioskop Online, bioskop21, BioskopKeren, Cinemaindo, Dewanonton, Drakor ID, DrakorIndo, Drama, DramaQu, Dunia21, DutaFilm, IndoXX1, Indoxxi, KorDramas, LayarKaca21, Layarkaca21 INDOXXI, LK21, LK21 XXI, Nonton drama, Nonton Movie, Thriller, Germany, Russian Federation, USA
Paul Greengrass
Forever (2023)
Bioskop Online, bioskop21, BioskopKeren, Cinemaindo, Dewanonton, Drakor ID, DrakorIndo, Drama, DramaQu, Dunia21, DutaFilm, Ganool, gudangmovie, gudangmovie21, IndoXX1, Indoxxi, KorDramas, LayarKaca21, Layarkaca21 INDOXXI, LK21, LK21 XXI, Nonton drama, Nonton Movie, Pahe.in, PusatFilm21, Sweden
Anders Hazelius
A Place Called Silence (2024)
Bioskop Online, bioskop21, BioskopKeren, Cinemaindo, Crime, Dewanonton, Drakor ID, DrakorIndo, DramaQu, Dunia21, DutaFilm, Ganool, gudangmovie, gudangmovie21, IndoXX1, Indoxxi, KorDramas, LayarKaca21, Layarkaca21 INDOXXI, LK21, LK21 XXI, Nonton drama, Nonton Movie, Pahe.in, PusatFilm21, Thriller, Malaysia, Singapore, Taiwan
Boon-lip Quah
74 min
Ghostbusters II (1989)
Bioskop Online, bioskop21, BioskopKeren, Cinemaindo, Comedy, Drakor ID, DrakorIndo, DramaQu, Dunia21, DutaFilm, Fantasy, Ganool, gudangmovie, gudangmovie21, IndoXX1, Indoxxi, KorDramas, LayarKaca21, Layarkaca21 INDOXXI, LK21, LK21 XXI, Nonton drama, Nonton Movie, Pahe.in, PusatFilm21, USA
Ivan Reitman
How to Train Your Dragon 2 (2014)
Action, Adventure, Animation, bioskop21, BioskopKeren, Celebrities, Cinemaindo, Comedy, Drakor ID, DrakorIndo, DramaQu, Family, Fantasy, Ganool, gudangmovie, gudangmovie21, IndoXX1, Indoxxi, KorDramas, LayarKaca21, Layarkaca21 INDOXXI, LK21, LK21 XXI, Nonton drama, Nonton Movie, Pahe.in, PusatFilm21, USA
Dean DeBlois
No More Posts Available.
No more pages to load.
