NKF

NKF - Ruby extension for Network Kanji Filter

Description¶ ↑

This is a Ruby Extension version of nkf (Netowrk Kanji Filter). It converts the first argument and return converted result. Conversion details are specified by flags as the first argument.

Nkf is a yet another kanji code converter among networks, hosts and terminals. It converts input kanji code to designated kanji code such as ISO-2022-JP, Shift_JIS, EUC-JP, UTF-8 or UTF-16.

One of the most unique faculty of nkf is the guess of the input kanji encodings. It currently recognizes ISO-2022-JP, Shift_JIS, EUC-JP, UTF-8 and UTF-16. So users needn't set the input kanji code explicitly.

By default, X0201 kana is converted into X0208 kana. For X0201 kana, SO/SI, SSO and ESC-(-I methods are supported. For automatic code detection, nkf assumes no X0201 kana in Shift_JIS. To accept X0201 in Shift_JIS, use -X, -x or -S.

Flags¶ ↑

-b -u¶ ↑

Output is buffered (DEFAULT), Output is unbuffered.

-j -s -e -w -w16¶ ↑

Output code is ISO-2022-JP (7bit JIS), Shift_JIS, EUC-JP, UTF-8N, UTF-16BE. Without this option and compile option, ISO-2022-JP is assumed.

-J -S -E -W -W16¶ ↑

Input assumption is JIS 7 bit, Shift_JIS, EUC-JP, UTF-8, UTF-16LE.

-J¶ ↑

Assume JIS input. It also accepts EUC-JP. This is the default. This flag does not exclude Shift_JIS.

-S¶ ↑

Assume Shift_JIS and X0201 kana input. It also accepts JIS. EUC-JP is recognized as X0201 kana. Without -x flag, X0201 kana (halfwidth kana) is converted into X0208.

-E¶ ↑

Assume EUC-JP input. It also accepts JIS. Same as -J.

-t¶ ↑

No conversion.

-i_¶ ↑

Output sequence to designate JIS-kanji. (DEFAULT B)

-o_¶ ↑

Output sequence to designate ASCII. (DEFAULT B)

-r¶ ↑

{de/en}crypt ROT13/47

-h –hiragana –katakana –katakana-hiragana¶ ↑

-h1 –hiragana: Katakana to Hiragana conversion.
-h2 –katakana: Hiragana to Katakana conversion.
-h3 –katakana-hiragana: Katakana to Hiragana and Hiragana to Katakana conversion.

-T¶ ↑

Text mode output (MS-DOS)

-l¶ ↑

ISO8859-1 (Latin-1) support

-f[`m` [- `n`]]¶ ↑

Folding on m length with n margin in a line. Without this option, fold length is 60 and fold margin is 10.

-F¶ ↑

New line preserving line folding.

-Z¶ ↑

Convert X0208 alphabet (Fullwidth Alphabets) to ASCII.

-Z -Z0: Convert X0208 alphabet to ASCII.
-Z1: Converts X0208 kankaku to single ASCII space.
-Z2: Converts X0208 kankaku to double ASCII spaces.
-Z3: Replacing Fullwidth >, <, “, & into '>', '<', '"', '&' as in HTML.

-X -x¶ ↑

Assume X0201 kana in MS-Kanji. With -X or without this option, X0201 is converted into X0208 Kana. With -x, try to preserve X0208 kana and do not convert X0201 kana to X0208. In JIS output, ESC-(-I is used. In EUC output, SSO is used.

-B¶ ↑

Assume broken JIS-Kanji input, which lost ESC. Useful when your site is using old B-News Nihongo patch.

-B1: allows any char after ESC-( or ESC-$.
-B2: forces ASCII after NL.

-I¶ ↑

Replacing non iso-2022-jp char into a geta character (substitute character in Japanese).

-d -c¶ ↑

Delete r in line feed, Add r in line feed.

-m¶ ↑

MIME ISO-2022-JP/ISO8859-1 decode. (DEFAULT) To see ISO8859-1 (Latin-1) -l is necessary.

-mB: Decode MIME base64 encoded stream. Remove header or other part before

conversion.

-mQ: Decode MIME quoted stream. '_' in quoted stream is converted to space.
-mN: Non-strict decoding.

It allows line break in the middle of the base64 encoding.

-m0: No MIME decode.

-M¶ ↑

MIME encode. Header style. All ASCII code and control characters are intact. Kanji conversion is performed before encoding, so this cannot be used as a picture encoder.

-MB: MIME encode Base64 stream.
-MQ: Perfome quoted encoding.

-l¶ ↑

Input and output code is ISO8859-1 (Latin-1) and ISO-2022-JP. -s, -e and -x are not compatible with this option.

-L¶ ↑

new line mode Without this option, nkf doesn't convert line breaks.

-Lu: unix (LF)
-Lw: windows (CRLF)
-Lm: mac (CR)

–fj –unix –mac –msdos –windows¶ ↑

convert for these system

–jis –euc –sjis –mime –base64¶ ↑

convert for named code

–jis-input –euc-input –sjis-input –mime-input –base64-input¶ ↑

assume input system

–ic=`input codeset` –oc=`output codeset`¶ ↑

Set the input or output codeset. NKF supports following codesets and those codeset name are case insensitive.

ISO-2022-JP: a.k.a. RFC1468, 7bit JIS, JUNET
EUC-JP (eucJP-nkf): a.k.a. AT&T JIS, Japanese EUC, UJIS
eucJP-ascii: a.k.a. x-eucjp-open-19970715-ascii
eucJP-ms: a.k.a. x-eucjp-open-19970715-ms
CP51932: Microsoft Version of EUC-JP.
Shift_JIS: SJIS, MS-Kanji
CP932: a.k.a. Windows-31J
UTF-8: same as UTF-8N
UTF-8N: UTF-8 without BOM
UTF-8-BOM: UTF-8 with BOM
UTF-16: same as UTF-16BE
UTF-16BE: UTF-16 Big Endian without BOM
UTF-16BE-BOM: UTF-16 Big Endian with BOM
UTF-16LE: UTF-16 Little Endian without BOM
UTF-16LE-BOM: UTF-16 Little Endian with BOM
UTF8-MAC: NKDed UTF-8, a.k.a. UTF8-NFD (input only)

–fb-{skip, html, xml, perl, java, subchar}¶ ↑

Specify the way that nkf handles unassigned characters. Without this option, –fb-skip is assumed.

–prefix= `escape character` `target character` ..¶ ↑

When nkf converts to Shift_JIS, nkf adds a specified escape character to specified 2nd byte of Shift_JIS characters. 1st byte of argument is the escape character and following bytes are target characters.

–disable-cp932ext¶ ↑

Handle the characters extended in CP932 as unassigned characters.

–cap-input¶ ↑

Decode hex encoded characters.

–url-input¶ ↑

Unescape percent escaped characters.

–¶ ↑

Ignore rest of -option.

Constants

ASCII: ASCII
AUTO: Auto-Detect
BINARY: BINARY
EUC: EUC-JP
JIS: ISO-2022-JP
NKF_RELEASE_DATE: Release date of nkf
NKF_VERSION: Version of nkf
NOCONV: No conversion
SJIS: Shift_JIS
UNKNOWN: UNKNOWN
UTF16: UTF-16
UTF32: UTF-32
UTF8: UTF-8
VERSION: Full version string of nkf

Public Class Methods

guess1(str) → integer click to toggle source

Returns guessed encoding of str as integer.

Algorithm described in: Ken Lunde. `Understanding Japanese Information Processing' Sebastopol, CA: O'Reilly & Associates.

case NKF.guess1(input)
when NKF::JIS
  "ISO-2022-JP"
when NKF::SJIS
  "Shift_JIS"
when NKF::EUC
  "EUC-JP"
when NKF::UNKNOWN
  "UNKNOWN(ASCII)"
when NKF::BINARY
  "BINARY"
end

 
               static VALUE
rb_nkf_guess1(obj, src)
  VALUE obj, src;
{
  unsigned char *p;
  unsigned char *pend;
  int sequence_counter = 0;

  StringValue(src);
  p = (unsigned char *)RSTRING(src)->ptr;
  pend = p + RSTRING(src)->len;
  if (p == pend) return INT2FIX(_UNKNOWN);

#define INCR do {\
      p++;\
      if (p==pend) return INT2FIX(_UNKNOWN);\
      sequence_counter++;\
      if (sequence_counter % 2 == 1 && *p != 0xa4)\
        sequence_counter = 0;\
      if (6 <= sequence_counter) {\
          sequence_counter = 0;\
          return INT2FIX(_EUC);\
      }\
  } while (0)

  if (*p == 0xa4)
    sequence_counter = 1;

  while (p<pend) {
    if (*p == '\033') {
      return INT2FIX(_JIS);
    }
    if (*p < '\006' || *p == 0x7f || *p == 0xff) {
      return INT2FIX(_BINARY);
    }
    if (0x81 <= *p && *p <= 0x8d) {
      return INT2FIX(_SJIS);
    }
    if (0x8f <= *p && *p <= 0x9f) {
      return INT2FIX(_SJIS);
    }
    if (*p == 0x8e) {   /* SS2 */
      INCR;
      if ((0x40 <= *p && *p <= 0x7e) ||
          (0x80 <= *p && *p <= 0xa0) ||
          (0xe0 <= *p && *p <= 0xfc))
        return INT2FIX(_SJIS);
    }
    else if (0xa1 <= *p && *p <= 0xdf) {
      INCR;
      if (0xf0 <= *p && *p <= 0xfe)
        return INT2FIX(_EUC);
      if (0xe0 <= *p && *p <= 0xef) {
        while (p < pend && *p >= 0x40) {
          if (*p >= 0x81) {
            if (*p <= 0x8d || (0x8f <= *p && *p <= 0x9f)) {
              return INT2FIX(_SJIS);
            }
            else if (0xfd <= *p && *p <= 0xfe) {
              return INT2FIX(_EUC);
            }
          }
          INCR;
        }
      }
      else if (*p <= 0x9f) {
        return INT2FIX(_SJIS);
      }
    }
    else if (0xf0 <= *p && *p <= 0xfe) {
      return INT2FIX(_EUC);
    }
    else if (0xe0 <= *p && *p <= 0xef) {
      INCR;
      if ((0x40 <= *p && *p <= 0x7e) ||
          (0x80 <= *p && *p <= 0xa0)) {
        return INT2FIX(_SJIS);
      }
      if (0xfd <= *p && *p <= 0xfe) {
        return INT2FIX(_EUC);
      }
    }
    INCR;
  }
  return INT2FIX(_UNKNOWN);
}

guess2(str) → integer click to toggle source

Returns guessed encoding of str as integer by nkf routine.

case NKF.guess(input)
when NKF::ASCII
  "ASCII"
when NKF::JIS
  "ISO-2022-JP"
when NKF::SJIS
  "Shift_JIS"
when NKF::EUC
  "EUC-JP"
when NKF::UTF8
  "UTF-8"
when NKF::UTF16
  "UTF-16"
when NKF::UNKNOWN
  "UNKNOWN"
when NKF::BINARY
  "BINARY"
end

 
               static VALUE
rb_nkf_guess2(obj, src)
  VALUE obj, src;
{
  int code = _BINARY;

  reinit();

  input_ctr = 0;
  StringValue(src);
  input = (unsigned char *)RSTRING(src)->ptr;
  i_len = RSTRING(src)->len;

  if(x0201_f == WISH_TRUE)
    x0201_f = ((!iso2022jp_f)? TRUE : NO_X0201);

  guess_f = TRUE;
  kanji_convert( NULL );
  guess_f = FALSE;

  if (!is_inputcode_mixed) {
    if (strcmp(input_codename, "") == 0) {
      code = _ASCII;
    } else if (strcmp(input_codename, "ISO-2022-JP") == 0) {
      code = _JIS;
    } else if (strcmp(input_codename, "EUC-JP") == 0) {
      code = _EUC;
    } else if (strcmp(input_codename, "Shift_JIS") == 0) {
      code = _SJIS;
    } else if (strcmp(input_codename, "UTF-8") == 0) {
      code = _UTF8;
    } else if (strcmp(input_codename, "UTF-16") == 0) {
      code = _UTF16;
    } else if (strlen(input_codename) > 0) {
      code = _UNKNOWN;
    }
  }

  return INT2FIX( code );
}

nkf(opt, str) → string click to toggle source

Convert str and return converted result. Conversion details are specified by opt as String.

require 'nkf'
output = NKF.nkf("-s", input)

Note By default, nkf decodes MIME encoded string. If you want not to decode input, use ::nkf with -m0 flag.

 
               static VALUE
rb_nkf_kconv(obj, opt, src)
  VALUE obj, opt, src;
{
  char *opt_ptr, *opt_end;
  volatile VALUE v;

  reinit();
  StringValue(opt);
  opt_ptr = RSTRING(opt)->ptr;
  opt_end = opt_ptr + RSTRING(opt)->len;
  nkf_split_options(opt_ptr);

  incsize = INCSIZE;

  input_ctr = 0;
  StringValue(src);
  input = (unsigned char *)RSTRING(src)->ptr;
  i_len = RSTRING(src)->len;
  result = rb_str_new(0, i_len*3 + 10);
  v = result;

  output_ctr = 0;
  output     = (unsigned char *)RSTRING(result)->ptr;
  o_len      = RSTRING(result)->len;
  *output    = '\0';

  if(x0201_f == WISH_TRUE)
    x0201_f = ((!iso2022jp_f)? TRUE : NO_X0201);

  kanji_convert(NULL);
  RSTRING(result)->ptr[output_ctr] = '\0';
  RSTRING(result)->len = output_ctr;
  OBJ_INFECT(result, src);

  return result;
}

Home Classes Methods

In Files

Methods

Class/Module Index

NKF

Description¶ ↑

Flags¶ ↑

-b -u¶ ↑

-j -s -e -w -w16¶ ↑

-J -S -E -W -W16¶ ↑

-J¶ ↑

-S¶ ↑

-E¶ ↑

-t¶ ↑

-i_¶ ↑

-o_¶ ↑

-r¶ ↑

-h –hiragana –katakana –katakana-hiragana¶ ↑

-T¶ ↑

-l¶ ↑

-f[m [- n]]¶ ↑

-F¶ ↑

-Z¶ ↑

-X -x¶ ↑

-B¶ ↑

-I¶ ↑

-d -c¶ ↑

-m¶ ↑

-M¶ ↑

-l¶ ↑

-L¶ ↑

–fj –unix –mac –msdos –windows¶ ↑

–jis –euc –sjis –mime –base64¶ ↑

–jis-input –euc-input –sjis-input –mime-input –base64-input¶ ↑

–ic=input codeset –oc=output codeset¶ ↑

–fb-{skip, html, xml, perl, java, subchar}¶ ↑

–prefix= escape character target character ..¶ ↑

–disable-cp932ext¶ ↑

–cap-input¶ ↑

–url-input¶ ↑

–¶ ↑

Constants

Public Class Methods

-f[`m` [- `n`]]¶ ↑

–ic=`input codeset` –oc=`output codeset`¶ ↑

–prefix= `escape character` `target character` ..¶ ↑